So what am I working on? Well, from a top level. The current end-to-end experience of designing and building custom manufactured products really sucks. I’m building BOM Quote Manufacturing with a vision to make it better.

The strategy I chose in starting BOM Quote MFG seven years ago, out of necessity to earn a living while bootstrapping, is what I call, “Factory First.”

By that, I mean quite literally, first, we built a fully licensed export CM factory in Shenzhen, China. We help customers by building and shipping their custom designed retail packaged products.

Doing all the offline stuff to get our private factory setup in China, took several years of work with a hefty dose of patience. Now, we are following up with custom software, to help us improve our service, efficiencies, and scale.

With that, nearly all my 30-hours per week coding goal over these last three years went toward building our core bomquote.com web app. It aims to wrap a solid layer of communication and processes, around our interaction with both customers and suppliers.

One of the problems we repeatedly face boils down to gathering and assimilating data which is available on the internet.

For example, data about pricing and stock status for electronic components, as listed on many websites which sell such things.

Fact is, much of the data that we need to move quotes forward, especially early in the product development process, is available on the internet. We need to automate that web data gathering.

So, I present Transistor, a flexible web scraping and data persistence framework. Transistor will serve as the nucleus of our own web data collection efforts, at BOM Quote Manufacturing.

Transistor supports using scrapinghub’s Splash “javascript rendering service” as a headless browser. It also supports their crawlera “smart proxy” service. Transistor uses well-known libraries like python-requests, beautifulsoup4, mechanicalsoup, and gevent. Scrapy is not required.

Frankly, at this point, you should probably just use Scrapy, unless you suspect you have some valid reasons for not using it as I did. But, further detailing Transistor and it’s use, is better left to the README and future articles.

Transistor started as a utility module inside our core bomquote.com web app. I worked on it about 32 hours and had written the first 2000 lines of code before I decided to break it out into a dedicated repository.

That first 32 hours and 2,000 lines of code, is the part in green in the chart above. And, below is how my time looks in Wakatime for the for Transistor repository, from the initial commit with that first 2,000 LOC on October 7th:

In summary, I logged about 220 total hours on Transistor in October. Factoring in about four days off from coding, that’s a little over 8-hours per day in daily average coding hours, for the days I worked.

In moving beyond hours tracked, Wakatime runs out of legs. I use other apps for insight, including codecov.io and codeclimate.com. Those two overlap a bit, in that Code Climate offers some feedback on test coverage, if you use it. While, I’ve always preferred to continue using codecov.io for test coverage, due to it’s UI.

But hands down, Code Climate is the best service I use to provide feedback on my code, other than test coverage. And, here is what Transistor looked like at the end of October, on the overview page, in Code Climate:

Clicking on the “Trends” tab will bring up a “Maintainability” page with a few different sections. The view defaults to the first section, “Technical Debt,” shown below:

The technical debt section reports, I started with about 50 hours technical debt on my first commit, in that first 2000 lines of code. But the first 2000 lines of code only took me about 32 hours. So, I coded 32 hours to create 50 hours of technical debt? Seems a bit off.

Over the following two weeks, total debt hours increased, while the debt to lines-of-code ratio decreased, as I committed an additional 2,000 lines of code. While only adding 20 hours more technical debt over that time frame.

So, what is technical debt? Basically, it is a measure of the estimated time required to resolve items that Code Climate surmises I should fix in my code.

For example, functions and class methods with high “cognitive complexity.” In my case, there were two “D” graded files for my initial pass at low-level website scrape logic, with about 300 lines of code in each file.

I knew that code was nasty when I wrote it, but thanks Code Climate, for confirming it. And, most of the 463 other issues are PEP-8 infractions for line code length. Should be 79 chars while I set mine at 88 chars. Sorry, not sorry.

Takeaway? I love the code grades and smell highlights. But, it also seems to me, they’ve overstated the technical debt workload. It should take me like one day, to clean up a majority of the noted technical debt issues. Definitely, not seventy hours of work.

Further, it is off by such an amount that, I’d hate to expose my team to those time estimates as anchor points. The code climate algorithm could use some management tuning. Shouldn’t blindly accept that number from the team.

Next, Code Climate has a lines-of-code chart. It shows, my first repository commit was on October 8th. And, it shows that in October, I committed about 4000 lines of code to Transistor.

In my first week developing Transistor, I had an open canvas. So, it was not super challenging to write 2000 lines of python code from scratch. But after that, it became more difficult to churn out lines.

This chart shows, I only wrote 800–900 lines, the week I worked 90 hours, Oct 17-Oct 24th. Looking back, by then, I was more constrained in my canvas and spent much of the week optimizing Lua scripts to reduce crawl time. The result was, my LOC per work-hour, dropped quickly.

Takeaway? Lines of code helps tell a story, but it isn’t the full story. Productivity gauged on LOC must be done in full light of the situation.