Software Engineer unravels the tech industry
The technology industry has grown from just one bubble to many millions of bubbles. Some may even go the extent to name the past two decades as the "technology boom", with various, remarkable technologies being invented, whether it be tools which we use but take for granted every day, to technologies in the making to solve and change the world for the better.
This industry is so heavily vast that sometimes it can be difficult to navigate through all the opportunities which it provides. Machine learning. Web development. Software. Hardware. A handful of areas which have been intensely useful for our understanding of the world around us and how people interact with each other. It's not just programmers or developers who can learn how to programme. Programming is a very valuable tool to have no matter what industry or career you wish to have.
We are joined by Joydeep Bhattacharjee, a professional software developer, who has chosen to discuss the values and characteristics which the tech industry brings along with it and also, his experience and advice which he has accumulated whilst working as a software engineer.
1 | Describe what you do on a daily basis as a software engineer?
Software Engineering is about empowering business leaders solve real world problems. That is has two primary parts - Problem Solving and Communication. So I will say that I spend around 50 percent of my time in solving problems and 50 percent of my time in communicating with the different stakeholders. Creating machine learning models is more about giving correct data and in an industrial environment it becomes important that the data is in the correct format. Hence creating and maintaining good data pipelines is of high importance in our day to day activities. This needs good knowledge of various data technologies, both SQL and NoSQL. MySQL and PostgreSQL and quite popular database engines in SQL world.
In NoSQL I would highly recommend MongoDB. We use graph databases to show relationships between data and so Neo4J is a good choice in that. Most of the real world data is messy and hence machine learning is both about creating good models as well as doing a lot of data preprocessing. A good scheduling platform is needed to handle large amounts of data and for that Airflow is a good choice. Lastly to create models both R and Python are good language choices, but python is a better language for a machine learning team as you can create both the models and the data pipelines in the same language and there is no context switching; the same team is able to do both.
Apart from work, it is highly important that a machine learning engineer or a software engineer not only works but is continually learning and is continually searching for new tools. So it becomes important that he or she is not working all the time, but devoting sufficient time for both research and study, and rest and recreation. I generally work for 8 hours a day, give around 2 hours to do some research and study and the remaining time to my family. Apart from that we take regular vacations once every few months away from work. These days there are various avenues of study and depending on the individual he can listen to podcast, see YouTube videos, buy books and attend workshops. I try all these methods.
2 | What made you choose a career as a software engineer?
I have an interesting story how I came to chose a career as a software engineer, but I don't believe it to be unique. I majored in Electrical Engineering at University and hence had to do some scientific programming in my university days which was mostly in C lang, I had also used matlab. Scientific programming is different, at least in the way we used it in our university, from software programming in that there was almost no focus on reusability and building abstractions and writing clean code. In software engineering, these factors are of paramount importance. I got to know that a lot later. In any case when I joined the industry, I tried learning Java, which was the defacto language at the time. The verbosity of Java repelled me. I am not trying to make rant against Java, I believe now that the language has a lot of merit, just that at the time the language didn't speak to me. The unnecessary amount of verbosity seemed meaningless to me.
I had liked the C lang, but the ubiquitousness of the Java lang made me feel that this is probably the way “industrial” programming worked and I didn't want to be a part of it. So I chose systems work in the company which was related to working in Unix/Linux type environments. After working in some projects I started working in a project where they were running a lot of Python scripts. Due to my work with Linux systems I knew that I could open bash scripts in a simple text editor. Since Python is a dynamic language I discovered that I could open Python in a simple text editor as well. That was kind of the turning point in my life. I researched about the language and found that Python is the gateway to this whole new world of various domains and it was not that verbose as well. I knew I had found my path.
3 | What do you love about not only being involved in technology but also your specific career as an engineer?
I love the fact that experimentation has become cheaper. I believe mankind is inherently curious but sometimes we are not able to do what we want due to cost involved in the experimentation. This cost of experimentation has reduced to a large extent. You can run complicated logical simulations using software. You can create worlds depending on specific rules like in the video game Portal. You can do complicated mathematical calculations and see how populations behave under specific constraints. If you have some business idea of selling a niche product you can just create a website and see if that idea has value. This was not possible before and flights of imagination was only possible in either pure mathematics or the visual and literary arts, but now software has somehow made it more “real”.
This is the same thing about my daily work. In my daily work we sit down as a team and brainstorm ideas and then go about testing them or build stuff to test them. The only real cost here is developer time and the biggest risk of a bad idea is that probably we will be building something that no one would want or use and we will probably waste some days or worst some months. That I believe is such an improvement.
One may argue that this is still not a perfect world, but I would say this is a lot better.
4 | What first grew your interest of technology and how did that influence your career options?
My interest in technology has come due to some interesting coincidences during the years that I was growing up. The school that I had grown up studying in had scarcity of resources so advanced math and sciences were only reserved for selected students. Fortunately, my teachers had faith in me and I was able to become get enrolled in those courses. This gave me the confidence that probably I am most suitable for the engineering sciences. Apart from that the political party in our state, in trying to get more favourable public opinion, promised that in case they come to power they would distribute computers in case they come to power. They came to power and I received a computer with public money.
This was a huge thing for me at that stage and we didn't have the resources to buy a computer. There was still a lot of years before I could get a working and reliable internet connection but still I was able to learn a lot using that government computer. I would say if I didn't have the good fortune of owning that computer in my earlier years, I would probably not have had the confidence to take a leap towards software development. I have come across a lot of people who will are intimidated by the device and see it as some sort of magic black box, although I believe that part is waning due to the current prevalence of apple and android devices. Probably the next generation would grow up to take the ubiquitousness of computing devices for granted.
5 | What is some advice you would give to aspiring students wishing to go into courses such as computer science and engineering?
I believe learning to code will become an essential skill no matter the occupation of an individual. In cases a student chooses a different profession the type of technical learning that he or she will become fairly particular. In case the student is interested in software development as a career choice there is a general confusion and challenge of decided what to learn. There are so many technologies and skills out there. I would say it's the skill of coding that is the most important skill to get better at. But even if a student focuses on programming there is still a lot of languages out there.
6 | In your opinion, why do you think technology plays such as significant role in today's society?
I believe there are three aspects of technology that has changed our society in profound aspects.
As I had stated earlier, the cost of experimentation has come down a lot. This has encouraged individuals and groups to invest more in experimentation and original research.
Technology has enabled automation over vast swathes of life. This has twofold effects. Traditional professions and job titles will become increasingly irrelevant thus forcing individuals to become more entrepreneurial and thus fostering more innovation in various walks of live. The other aspect is that there has been rise of powerful organisations such as Facebook and Google. The value that such companies are able to bring to the table stem from the amount of automation that they are able to bring to the table for complex information processing.
The third profound change that technology has brought about is a bit more subtle and can be specifically attributed to the internet. The internet has brought about a decentralisation of information and ease of communication on a scale not seen before. Communication is to learning and progress as oxygen is to fire. An interesting theory in this case is that the Dark ages descended on Europe in the middle ages because the Roman empire had fallen and there was no one to maintain the good roads and communication lines to maintain the once good roads. Its hard to come up with good ideas alone.
7 | Do you have a reading list of books that you think would be helpful for students wanting to take up computer science and engineering?
Programming Pearls by Jon L. Bentley
The ALgorithm Design by Skiena
Clean Code by Robert Cecil Martin
Coders at Work by Peter Seibel
Learn Python the hard way by Zed Shaw
Learning Python by Mark Lutz
The Elements of Statistical learning
Artificial intelligence a modern approach by Peter Norvig
Find Joydeep at his podcast | Podcast
Read more of his articles on Medium | Medium