Some day in the near future, the marketing department will wonder why so many people were curious about all of their products after browsing Organic Ground Beef[0]
Indeed, he spelled it the right way, which is the wrong way. He should spell it wrong, which is the right way. :)
I think it is funny that this misspelling hasn't been fixed after all of these years. It was typo'ed in the original http spec in 1996: https://en.wikipedia.org/wiki/HTTP_referer
If the web server is following RFC 8969, it will treat "referrer" as "referer" and throw a 397 TOLERATING to let you know you should change it to the latter on your end. See Section 3.
There are regional variations in the use of doubled or single consontants:
UK: travelling, signalling
USA: traveling, signaling
The double l serves no purpose in these.
However, there is no such UK/USA split in "referring": both use double "r".
There is a phonetic reason for that which is that the second syllable has the emphasis in "refer", unlike "traveling" and "signaling" where the leading syllable has it.
The double "rr" indicates the stressed syllable, without which "refering" and "referer" could be misread with emphasis on the first syllable, like "buffering" or "suffering"/"sufferer".
A native speaker with good intuition for spelling would never write "refering" or "referrer", even if they find "signaling" acceptable.
I’ve always held on to my grocery receipts for the last 8 years. I took pictures of them but when I moved a box got water damaged so now I only have like the last 3ish years.
Is there any open source software that I can use to transfer these receipts into a useful csv?
I have an idea for a few interesting data visualizations as I’d often buy the same things every week. Grocery bill went from like $70 to $150 with not much changes from what I can tell.
I run paperless on an Unraid server at home and it works really well. It has "machine learning" (based on a model that you host yourself that grows as you use it). It has good search, impressive OCR, and generally works really well.
My only complaint is that it lacks a good organization workflow. I have a shared network folder and any file (image, pdf, etc) that you put on that folder gets immediately consumed into Paperless. This happens almost immediately. I have a printer/scanner that allows me to scan to a SMB network drive. So I configured any scans from it to go to that shared folder, which makes integration really nice. I also use GeniusScan on my phone to scan to the same network drive (which requires pro, which is ~$16 a year I think). Genius Scan can save locally to your phone and upload later when you get home, which makes for a good workflow. The problem is that once it gets into Paperless, there's not a good workflow for reviewing and labeling the file. I have been meaning to sit down and provide a contribution to the open source project to improve this, but haven't found the time to do it yet. This is the biggest weakness of the project imo.
For those that have never used paperless. The naming may confuse you. It started off as an open source project named paperless. Then it got abandoned and a team picked it up to update it and make it more modern, and they renamed it paperless-ng (for angular I assume, the new frontend). Then that project lost momentum, so it was forked again and is now paperless-ngx which is the current iteration of it. It currently has a very strong community and gets good updates.
I'd look into a document scanner for ease of use. They even have ones that auto loads, so no more waiting around. With that said, if you purchase a scanner, it probably already has proprietary OCR, and they have auto feeding ones for many documents. I foolishly bought one not knowing auto feeding was an option.
https://youtu.be/fi0ZhTFaW7w
I bought a brother 2 sided one since it had Linux drivers.
hmm, IDK if a scanner would help me. I already have pictures of my receipts. I might have to do more research because I feel like there's gotta be something out there where you can just show images of receipts and have it generate a csv of the data.
I'd even pay a decent amount to do it. After doing some more research it seems like MS Office might handle this workflow too in Excel (convert receipt picture to csv data).
I have attempted this and the biggest issue is that sometimes the receipts use codes hard to understand. And the codes will change from store to store.
If you're lucky, you won't need to go to a grocery store and determine what a code means, you will only need to map the code to an actual item you bought.
That’s perfectly fine for me. I can map the key items myself, the hard part is I don’t want to devote a solid 120+ hours manually creating the CSVs for 150 receipts.
I do the same with WholeFoods receipts. The pipeline is:
1. Scan to FTP dir to TIFF
2. Nightly job submits image(s) from the dir to Veryfi (their free tier is enough and they looks like the best for receipts OCR)
3. Save that raw JSON, enhanced JSON (fix occasional mis-attribution for discounts, calculate unit price), and CSV. Filename is a purchase date - correctly extracted by Veryfi.
4. Render with bash + gnuplot.
5. TODO: store into some DB and render with Grafana or something.
Given that these prices are going into a database, I was hoping that you could click on an item and get its pricing over time (not a referral link to the Trade Joes web site).
Sorry can't edit it now but you are correct and I wasn't thinking "that" kind of referral :-). Easy to see that interpretation though, will have to remind myself to use "links that reference the product".
FWIW cmoog, I think it is pretty cool. I've always felt that good surveillance on retail prices with specific product information (like weight/servings) would be a solid data set for economists and people like me to understand "real" inflation, "greed" inflation, and general product pricing trends.
I love the idea of pricing over time. It seems like things jump around in 50 cent increments quite a bit. Would be interesting to graph it. I wonder if you'd be able to gather enough data before TJ's decides to implement better endpoint security, though. Having tried to do something similar with airline website data, they sure can make it challenging.
My radical hack idea was to start an online newspaper and to contract with the local super markets to run a copy of their weekly insert. Fill the paper with local news and collect price data over time which could be marketed as a product to people who were interested in the data.
The most appealing part of this hack is that the advertisers weren't the customer, they were the product. :-)
Open food fact recently launched Open Prices (https://prices.openfoodfacts.org/). It's currently crowd-sourced instead of an automated crawling, but prices are localized in space and time which could lead to intersting results.
This will lead to an open database of food product prices.
It's a neat idea, but I think you need some automation to make it useful over a long period of time. There's a website to do track gas prices, and they just change too much to keep updated.
Localized pricing just adds noise to data. Intermittent updates combined with possibility of input error also creates issues.
Retailers that utilize price zones typically have the baseline price that drives the prices for each price zone (e.g. set prices in California to be 10% more than baseline). Getting the baseline price in an automated way is the ideal solution.
Pleasantly surprised to see about as many price reductions as price increases. Also funny that they marked up the price of roses ahead of Valentine's Day!
Depends. Sometimes turkey prices drop before thanksgiving, as the producers plan their growth cycles around expected thanksgiving demand, and use freezers to meet demand. For example, see the 2022 august peak in poult placement in: https://downloads.usda.library.cornell.edu/usda-esmis/files/...
The products often become loss-leaders (or close to it) to draw people into stores, so perhaps the interesting thing here is that valentines roses have a different pricing pattern -- probably because, unlike turkey, they don't freeze well, so supply is much less elastic.
Except for a the obvious counter examples. Like food in general very in demand and very cheap. Housing very in demand but very expensive. So it’s really more about who decides the prices than what.
Econ 101 explains those counter examples as well: Food has high demand but lots of competition and easy substitution (rice expensive? buy pasta). Housing supply is limited (by location but often artificially via regulation) and substitution is difficult (have to move).
People are not economically rational. Most people don't buy pasta when rice is expensive. Econ 101 explains rational actors in a vaccum, not gluten free, asian culture, anti Italians sentiment, or hatred of Monsanto.
> Most people don't buy pasta when rice is expensive.
They would if rice prices went through the roof, which they won't because people who sell rice are not stupid.
Econ 101 certainly explains why you can actually buy gluten free products now (supply for demand) and why gluten free products are usually more expensive (niche market).
My ECON 101 prof talked about the limitations of models and theories based on assumption of rational actors in the 1980s ... I have no idea why everyone thinks this is a gotcha.
I don’t know where you took Econ 101 but mine definitely covered how things like subsidies and protectionism perversely impact market forces. You’re just blaming economists for bad policies at this point.
How many recessions has "Econ 101" predicted? Economics is a "soft science", on the same tier as psychology which people love to ridicule. Understanding how rational actors behave in a microeconomic sense doesn't help you when you're talking about irrational actors in the macroeconomic sense
The basics of Supply and Demand, Price Elasticity, Inflation, Monopolies, etc. etc. don't require humans to be perfectly rational actors to be useful concepts.
Yes, Economics can't predict recessions or the next Beanie Baby fad, that doesn't mean supply and demand is nonsense.
“Demand goes up so price goes up” is true ceteris paribus. For instance, if the FDA banned Ozempic tomorrow, you would expect the price of food to go up because many of the Ozempic patients would return to their previous consumption habits. But there have been a thousand other things that have happened which adjusted the price up and down, and most of those events in the past 200 years made food cheaper instead of more expensive, because the world increased the amount of food it was able to grow a lot faster than people increased the size of their appetites.
In fact, this actually demonstrates another effect: as something becomes cheaper, people will consume more of it. Strictly speaking, demand is a curve where quantity demanded is inversely proportional to price; if “demand goes up” we usually mean the entire curve shifts to the right. There’s also a supply curve: quantity supplied is directly proportional to price. The intersection of these curves is the market price; nobody “decides the prices” unless you have a monopoly or cartel or government interference.
Housing, incidentally, is a classic example of government interference.
It doesn't really make sense to talk about a single product from a single supplier when talking about supply and demand. You completely leave out the competition side that drives the whole system.
A single example of gravity not working means the theory of gravity doesn't work, unless you mean that economics should not have scientific scrutiny.
Economics is in a dire state of proving theories right. It only accounts for perfectly rational actors. A famous nobel prize winner was proven wrong in the same year or so and he said his graphs didn't account for it.
In practical terms, yes. Adjusted for inflation, a modern iPhone costs roughly the same as the ones in prior generations, except now you get a lot more bang for your buck with better battery life, improved camera, etc.
iPhones aren't a monopoly (yet): they have to compete with Android phones still, plus Apple probably prioritizes increasing their marketshare and retaining customers over maximizing profit from iPhone sales. iPhones generate a lot of other profits from things like the app store, not just from the phone sales, so pricing the phones maximally would hurt those.
This caught me completely by surprise. I never would have imagined that a grocery chain, even Trader Joe's, would have a publicly-accessible API endpoint for its current product catalog.
I wonder how the author even found the endpoint! Is this at risk of Trader Joe's noticing and moving to require an auth token? I can't imagine the openness being intentional, at least outside that brief period of techno-optimism in the late 2000s where it seemed like everyone was offering data sources to build into your Twitter bot.
Possible secondary application: could this GraphQL endpoint potentially be used to determine when a product is being discontinued?
Trader Joes just doesn’t really compete on price except for a few loss leaders like milk and eggs. Most people I know who go to TJs treat it as their default store because of the shopping experience. The stores are smaller but well staffed, have a well curated and consistent product selection, and are just more pleasant than other grocery stores.
I don't have any hard data to add, but on the face of it this just seems wrong. Their cheese is famously good-for-price / cheap-for-quality, and I think their fresh produce is very competitive as well.
If your comparison is Whole Foods or one of the big grocery stores like Ralphs or Albertsons, sure Trader Joes looks competitive.
But if you compare it to stores that actually compete on price, it's overall much more expensive even if some items or categories are well priced. I.e. Grocery Outlet or Winco or Aldi (TJ's parent company?) or any number of ethnic grocery stores like Zion or Ranch 99 are much cheaper. YMMV but here in Southern California, there are probably a dozen different chains that compete on price that blow it out of the water.
Is this for one store or all stores? It's commendable that you posted your code, but a minimal README would be appreciated.
Looking through your code, I see that the default store is Chicago South Loop (701). This would be helpful information to include on the website displaying the results.
That's a misconception. Prices are not the same across stores.
I live in Los Angeles. Many times I've shopped at the TJ's in Silver Lake and one of the TJ's in Pasadena on the same day. Most prices are the same, but on many items the Silver Lake store is consistently 5-10% higher.
I've also shopped in midwestern TJ's and noted that the prices were generally lower than LA.
Products differ significantly, too. Items with the exact same name and packaging can be totally different regionally. For example, "Sonoma Chicken Salad" used to be a favorite of mine here in California. The Iowa version was disgusting, with roughly twice the mayonnaise, fewer nuts and grapes, and 3x the sugar.
I've noticed a similar thing with Trader Joes in PNW vs California. The produce selection was different, the pre-made foods (like the salads, wraps..etc) were somewhat different, meat selection and quality was also different.
Part of it is that TJ's used to be much more about the 'one-off buy of a weird but tasty product'. They would find a product they could sell, buy as much of it as was possible, and sell through it, never to order and sell it again. Over time though, it grew to be the store where people went for basic staples, and so the way they sourced products probably changed to a more traditional model that grocery stores use, where many of the more perishable products are regionally sourced.
So what you experienced with the Sonoma Chicken Salad (which, I commend your appreciation, that used to be a favourite of mine to get for lunch) is likely a result of them just being completely different products made in different places by different companies.
Trader Joes in the 90's and early 2000's was a cool quirky grocery store to pick up some fun stuff and good wine to round out the weekly grocery shopping. TJ's in 2024 feels like Kroger standing on Whole Foods shoulders wearing a trench coat.
They list prices on their website even without a local store selected. And then picking a local store in different locations, I can't find any prices that change.
I mean, obviously there might be exceptions. And I assume local produce varies, the same way it varies in every supermarket not just by the season but by the week. Most fresh produce isn't even listed on their site, and things like fresh salads are going to be based on local produce prices. (E.g. this daily price tracker doesn't have any entries for apples of any kind, for instance.)
But I can't find any evidence of any Trader Joe's products (whether frozen or snacks or jarred or bakery) having different prices between stores. Which is what I meant -- the stuff on their website. But it's good to clarify the difference between that and fresh produce.
(I could always be wrong, but you can find it repeated all over forums that Trader Joe's prices are the same everywhere, and they are in my experience as well -- it seems to be "common knowledge".)
wow I never knew that. Is there an official source stating that somewhere or it's common knowledge somehow or something people at the stores repeat to customers? I never noticed this before since I don't think I've ever gone to two different locations on the same day.
Yes, you make a good point. Although I suspect there may be regional differences in price, I haven't yet run the diff on that. Should be simple enough for me to allow the user to select their regional store location.
If you don't mind sharing, how do you find their API? I don't understand graphql that well and I've been trying to play with https://www.traderjoes.com/api/graphql to no avail. Cool project, github star achieved.
No. Liquor can only be sold at liquor stores, which close at 9pm Mon-Sat and are closed on Sunday. You also cannot ship liquor from another state. You can, however, import from another country. So if you want some rare bourbon you have to import it from overseas after it has been exported from the US. It's a real catch-22 situation. Or you can enter a lottery at a liquor store, which is often restricted to customers that spend a lot of money annually with that store.
Beer and wine can be bought at grocery stores, though on Sunday you cannot buy until at least noon. At 11:45 on a Sunday morning you will often see a few people waiting near the checkout with a cart full of groceries and a 12-pack.
The US is really bizarre about alcohol, and it varies wildly between states. Many states don't allow liquor sales in grocery stores, and only in either specially-licensed liquor stores, or in state-owned "ABC" (alcoholic beverage control) stores.
YES! thank you for doing this, I have been curious about some of the stuff I buy often and I felt like over the last year or two, things have climbed in prices a lot compared to the normal inflation price hikes.
Open nodes at stores could help BLS & commercial orgs doing regular price data gathering. I work on supplier pricing dynamics and for commoditized products freeing your pricing data is powerful.
This is cool, but with the apparently random ordering it's just about impossible to find anything. Add sort buttons? A description of why this ordering is used?
I think most non-perishable goods are the same across stores. There are regional differences between some perishable goods based on where they come from.
This is shredding close to HNs "please no political battle", but I'll try to take this as neutral as possible.
You are suggesting these are stores people should avoid for labor-unfriendly practices. Assuming that people are aligned with you on that value set, I don't think a sarcastic comment without actual recommendations, on a post about a tangentially related project, is going to move any needles.
And so, assuming the mods let us live: What are your recommended alternatives to TJs? TJs is, for better a worse, one of the few grocery store options with both decent quality and not entirely ludicrous prices. This project makes it even more enticing, because you can finally look up prices directly and see price history.
If there's another store that offers all that, I think a lot of people would be interested, independent of "why".
Didn't work at Trader Joes, but at another grocery chain for a while as a store buyer. I had essentially no control over pricing unless we had a bunch of backstock we had to move quick to avoid expiration. In those cases we had some level of store-level autonomy to "price to move." That being said, it was heavily tracked and if anyone was doing it too much I'm sure there would be consequences of some sort.
Besides that, we'd get updates from corporate with a list of new price tags we'd print out any time they changed something (100% with regional fluctuation baked in, but not at a store level).
I think there are enough grocery apps that it is better to show price data than not. Though I'm certainly baising that off how I shop rather than a general economics / game theory POV.
I regularly use the Ralphs app to cross shop while I'm at Trader Joes. They are only 2 minutes walking apart, so I normally start at TJs. However, sometimes I end up at Ralphs first and now having this data it could lead to an unplanned trip to TJs to save a few bucks.
[0]: https://github.com/cmoog/traderjoes/blob/ea2da58a84d3a04e28f...