Is writing a performant IB engine really that difficult? Would a stable one be possible if anyone other than pajeet neets attempted it?
Imageboard software
Other urls found in this thread:
github.com
lynxhub.com
github.com
en.wikipedia.org
github.com
en.wikipedia.org
github.com
github.com
gitgud.io
urbandictionary.com
sys.8ch.net
8ch.net
github.com
github.com
github.com
twitter.com
I asked about same thing in another thread.
Are you interested in doing it?
Are you interested in running performant IB or just writing software?
I am thinking about trying to write high performance IB, but I am not sure that I have social skills needed to run IB.
Why is it so difficult though? What is the main bottleneck? Are the people who did it so far just incompetent?
No, not really.
Then why aren't there any?
c h e c k m a t e
Why is it so difficult though?
Most web software has issues with performance. Most companies solve it by throwing money at hardware instead of improving software because good developers are expensive.
Chans do not seem like websites that are earning a lot of money so they cannot just throw money at hardware and survive while doing it.
What is the main bottleneck?
Currently, main bottleneck is that programming language used (PHP) and framework used (Laravel) is very slow.
According to people in other thread it takes PHP+Laravel 98ms just to start processing things.
I have not checked source code but it probably uses ORMs extensively and ORMs are very slow.
Another thing is that template rendering in PHP is painfully slow.
If chan becomes really popular then network IO will become a problem.
Are the people who did it so far just incompetent?
I would not call them incompetent. They followed some 'web programming' best practices which, unfortunately, are designed to make web programming accessible (to lower developer wages by sucking everyone into development) and they had to sacrifice performance to achieve it.
Can someone provide a flowchart/approach of the usual board?
I never really did webdev, just application programming so please spoonfeed me.
You need a bit of code to fetch threads and posts from database, and also bit of code to put threads and posts to database. That's basically it. There are of course nuances, like sanitazing user inputs so you don't get sql injection or captcha implementation, but otherwise it's all pretty simple.
You shouldn't be sanitising inputs, it should be abstracted in the first place
lmao what?
Yeah, user input should be abstracted into emojis
It is possible to do SQL injection using emojis.
Professionals probably don't want to deal with the various issues that go with running a chan for a userbase that's disrespectful and outright hostile towards you no matter what you try. Namely, CP spam and the fact that no advertiser will get within 100 feet of a chan (other than penis pump ads maybe) causing them to be massive money sinks.
Then why did Jim buy Holla Forums?
Because he had to feed the cripple to the pigs.
he had fun getting ownership of a jap site and selling users info, so thought to emulate the same in the US.
Yet with a competently implemented caching layer, 99.99% of requests shouldn't touch PHP or laravel.
When someone looks to acquire or run a site like this they are either a user who cares a lot about it, or they are an autist/jew looking for a new play thing. Guess which Jim is.
Assuming you had a very respectable owner, would it be bad at all to have a chan that requires you sign up to post? That would seriously deter CP autists and spam.
I'm a programming noob (took an intro to c++ course) who wants to make an imageboard in golang.
What textbooks should I download off of library genesis? I guess I need to learn:
Also, if you were making an infinity-style imageboard in golang, what would YOU use the goroutine concurrency feature for? Would you give every board its own goroutine? How about every active discussion thread?
Use Futaba's software. It's the most no-nonsense one out there and it just works.
where is that hosted?
No, two exist and do it well:
Werchan is based off werc, it is suckless and fast.
github.com
Then there is lynxhub, a library-less node app(no npm hell) that works perfectly well.
lynxhub.com
The only other way you can go is with a C app, C + Redis would be very good and satisfy the chan coding philosophy, otherwise you will just get bantz'd to death about choosing anything other than C.
Used to be hosted on 1chan, but that went down. So unfortunately I don't know.
Your narrative is kinda blown by Hiroyuki currently jewing the 4chan userbase.
Isn't it a gorillion years old?
doesn't this make Holla Forums Yattsuba?
...
Have to switch to HolyC and TempleOS for decent speed. All those abstraction layers are slowing things down.
The Tinyboard family is just fine. KusabaX isn't that bad either.
That's just a standard web forum, if you have to create account for posting.
you should use the existing Vichan+Infinity or Infinity Next IB software.
Don't fall for Stephen Lynxshit's lies.
IM SICK AND TIRED OF YOU PEOPLE DOUBTING ME, LYNXCHAN HAS BEEN PROVEN TO WORK OVER AND OVER AGAIN
FUCK NEXT AND ALL THE OTHER SHTIBOARDS
How many requests per second per core do you get?
Is it?
For me the core of a chan is the anonymity along with the general setup of the website.
I don't feel like needing an account changes much. Mods are the only ones who would be able to see any account name just like they can see our IP hash.
thats it. i am writing my own message/image board. in rust
One of the main reasons I like these chans is because I don't need an ID. I never had to create a username/password like I had to on www forums. Quite frankly, I'm not interested enough to do that anymore.
Remember when Holla Forums was saying "they could just tell" nextchan was more professional in code style and would of course be more robust.
Lynxchan had "ugly" variable names.
You've got at least 2 boards running your software now with no issues meanwhile nextchan is on fire.
Why retorts do you hear most about not choosing your software? I haven't seen a good argument on Holla Forums about it.
Said no one ever. Stop pulling shit out of your ass. It's joshcode.
Except that was exactly what was said, people only started saying lol joshcode after the changeover burned into the ground.
thats it. i am writing my own message/image board. in python
Consider contributing to an existing one instead. uchan is written in Python.
...
First, I leaked my trip.
Second, the ugly variable name was stupid from the beginning because the retard was looking at my front-end code, not the engine code.
Third, the most common reason I hear is "I am happy with my current software".
The second most common is "NODE IS SHIT, MONGO IS SHIT, KYS LELELELELE".
...
Remember when Holla Forums was unusable and full of pedo spam for the better part of a year? And how long has nextchan existed?
I'm preventing the creation of additional Python imageboard software :^)
You're going to have to do that unless you want to constantly reinvent the wheel. It doesn't look particulary spaghettiish to me.
fast
memory and thread safe
significant whitespace
no i think i will use rust
I was talking to someone who wanted to write an imageboard in Python, so recommending an imageboard written in Python seemed like the logical thing to do.
thats it. i am writing my own message/image board. in assembly
Fuck off prrpx
thats it. i am writing my own message/image board. by teaching NN how to write one
because zilion implementations that people are making give me more than enough data to do it
python is overkill for a web backend.
i am not prrpx.
thats it. i am writing my own message/image board. in bash
thats it. i am writing my own message/image board. in rust
there is no point in that, writing an imageboard is nothing but a waste of time.
...
it's the guy that made Clover.
What is Clover?
i dont care. i'm going to do it. in rust
A really nice 4chan Android app that Exodus (maybe the first Holla Forums app) is based on.
Leave it be user, you seriously need to go through programming fundaments first, then modern web service architectures, then learning Go and 'the Go way'.
You would give up after 2 weeks.
The core problem to solve is how to make an imageboard that makes money. You'll find the Jews banks control speech by limiting who can process credit card transactions. Antisemitism and delicious cake, while legal, cut you off from income as no one knows how to pay for things with real money anymore and the banks won't work with you. If they were profitable, real programmers (not you or Josh) would make great IB software on well-run servers.
i will do it for free. in rust
Ok, user.
...
Have you started working on it or are you just posting shit here?
i will start tomorrow. going to bed in a few minutes.
What's the best Android app for this site? I was using overchan until the pig farmer broke everything.
Battery life and UI. I can shitpost 10x longer and easier.
dashchan
I use Ouroubouros. It works. I also used Overchan in the past. I don't know if it has been fixed yet.
good luck with it, you will regret it but you have been warned.
I meant that I don't know if Overchan has been fixed yet. I know Ouroubouros has.
FUCK OFF PSYCHO LYNXSHIT
GO SUCK ANOTHER BRAZILIANS COCK
MONGO IS TRASH
wow rude. where is your imageboard software?
I'm just going with BASIC. Should work in CP/M and everything.
Doesn't lynxchan lose data in a million different cases? Pretty embarrassing.
You can call it GOTOchan. Maybe put some nice Dijkstra quotes in the default banners.
so how would you use concurrency in an imageboard?
Meanwhile Nextchan just added document caching.
We fast now.
No, it's more like you are a fucking retarded brazillian who can't even read the MongoDB docs.
Holy shit, you don't
I mean, you "use" it, meaning that the library you're using for HTTP routing is spawning a goroutine every each time an HTTP request would be accepted
See gin-gonic, fasthttp and fasthttprouter
You really need to study some backend development, focusing on REST architecture
...
what?
Spoiler: async behavior with pseudo-synchronous language features is always a mistake. Webdevs need to learn to accept this and just write async code.
pls elaborate
No but reimplementing all of the features people expect in a short timespan is difficult. Holla Forums loves to shit on Josh, for example, but no one practically helped. Meanwhile, there exists nextchan that's getting document caching set up on Infinity Next.
I love me some Redis but I have doubts about its scaling.
...
Lynxchan started one month after next. Just saying.
i purged all cache and logged out so there should be no more admins until I fix it i'm gonna remove varnish tomorrow
Also
Except that people that gave him money so he could do NOTHING but work on next.
And all those contributors that he asked for their doxx.
And odili that helped him big time testing it and adding cache and tries to help people working on next up to this day.
m712 doesn't understand the caching he's doing. They were on the right track.
Yeah and it's a piece of shit that's slower than a PHP web framework at templating, not to even mention all the inadequacies in the templates themselves. Calling app namespaces in the template that should be done in a controller, for one.
What's your point, lynx?
Making less than McDonalds for a years worth of work and reimplementing something that took dozens of people 6 years?
Thos inadequacies are in Next, for the record. Slow your nodejs slinkies are slower than PHP with bottlenecks in the way various pages are templated, according to profilers that Josh used.
How sad.
Other than the performance issues which should be obvious, think of lifetime management. This is where things get really ugly.
If I want to cancel a synchronous emulation of an asynchronous process (which is very frequent in real code, imagine writing a browser where the user starts loading a page then navigates away before it finishes for example), the point at which it is actually canceled is either poorly defined or requires adding a bunch of yields into the code. In C/C++ this really isn't possible to do right due to how broken thread cancellation is although people try anyway and 'good' code uses a signalling loop with a way to interrupt it. Go requires similar with channels. This requires also handling the possibility of races and having to use barriers if you need to be sure things really stopped. It turns into a complex and buggy mess very quickly.
Now consider just writing it async to begin with. Every point at which the code returns to the event loop is a safe yield point, these are the only points where you could attempt to do so anyway so you don't even need to think about it, and the process can be disposed of and cleaned up without any complexity like signalling loops, locks, barriers, etc.. It's harder for programmers who aren't used to state machines to think about but once you are they're much more flexible, easier to keep bug-free, easier to debug as they require no special tools, and don't require any special language support (can easily be called by any other language).
Yeah, remember that time endchan shat the bed because a half dozen pedos migrated to it?
Oh, wait, it took hundreds of gooks without a hick while nextchan couldn`t support a handful of pedos.
Remember the time that you admitted that your web application templates slower than Next?
And that you're still attempting to compare something that has no document caching yet, to Lynxchan which does? And that you're an incompetent brazillian for even comparing the two in that context?
Let alone the fact that you can't seem to read the MongoDB docs for GridFS. Specifically the notice on atomicity.
Anyways, now that the faggot cat has left:
Josh was railroaded by a crippled gremlin and his pigfucker handler. I've lost all respect for hotwheels after I realized that what he was basically doing was attempting to embarass Josh into surrendering his copyright, and attempted to rip off the hundred or so users who donated in the process by making him surrender it to Jim, whose conditions were that it be non-free.
But hey, what do you expect from a company in a third world asian shithole that commits petty copyright violations (attempted to assert sole copyright on their Overchan fork) and has the fucking balls to rewrite git history instead of admitting the mistake?
Also, not just committed a petty copyright violation, but then redistributed it on the play store.
And still haven't fixed the captcha.
Embarassing.
Ok, I know all of that from the beginning.
What I meant was: "why the fuck is that important now?"
I mean, we are talking about a completely different things
Is that a copypasta that you have ready to use with your other autistic friends?
sry, *knew, *thing
Because you asked. Why did I bother.
Yeah but you started with an unrelated statement in the first place, I wanted to know if I was missing something
Has anyone benchmarked Lynxchan with cache? How many requests can it serve?
Lynxchan already has a cache built in, it caches using the GridFS file abstraction ontop of MongoDB.
But it serves less throughput than Varnish or NGINX serving static HTML files using spinning rust and the kernel cache.
already done nigger github.com
...
250 requests per second per core more or less.
pls leave
What?
It`s what odili told me.
pl.vichan.net + 8ch.pl run on a 16 eur dedicated server. pic related is top.
How many requests per second do popular chans actually need? Is there any point in making zilion rps imageboard software or 250 is enough?
It's pointless to worry about this too much, unless your goal is to be a popular website, but that pretty much guarantees the content will be dumbed-down consumerist shit (a quick scan of 4chan's /g/ catalog is enough to demonstrate this).
I'm more interested now in what can be done in "underground" networks like Freenet and such, where the storage, bandwidth and latency are most limiting factors.
Thank you for giving good reply. I thought that it is important because all those chans started deploying caches to improve their rps numbers.
Why use a library when you can make it yourself? I got this textbook Web Development With Go off of libgen, and I'm going to go through it.
Well then that's one way to use goroutines for basic stuff, but what about other features? These days, people want live-updating boards and threads and shit. Wouldn't concurrency and websockets be used for that? Sure I don't know how this shit works, but that's why I'm asking. I think the biggest problem with 8ch's servers for a while was the IO, so if I made an imageboard software, I'd want to be keeping live threads in memory, as threaded processes. Wouldn't I?
so what language handles async in a way that you like
people were only donating for Holla Forums.
holy shit guys. i just looked at a few other imageboard softwares (lynxchan, infinity next, µchan, vichan) and they are all complete garbage.
my chan in rust will be fucking amazing.
btw i need a name and a logo
A fork is in progress called Nextchan, see: gitgud.io
If you're actually serious about this, please consider the AGPLv3, other licenses allow people running an instance to close up the source code to the community and give nothing back (if they've actually made modifications). Jim Watkins does this and he's harming everyone including himself. This sort of behavior has been screwing up imageboard software progress for ages. Imageboard software is not like other software, there are no great big corporations shitting out tons of sourcecode by side effect to the community, there is no backing, there is just the people running the websites and the people who make the software.
what about adding to rustchan?
The majority of people don't care about open source, they just want something that works. Look at 4chan, a closed source botnet yet the open source activists of /g/ love it and can't get enough.
I can't say what would be best for Holla Forums in this regard though.
have you looked at the code? it is irredeemable shit
/g/tards don't give a fuck about libre software. just look how much GPU wars shitposting there was without any discussion of linux drivers.
vast majority of the open source activism is just virtue signaling.
I fully support this notion.
Furthermore, I second AGPLv3 and propose notabug, gitgud or some other github alternative.
gtfo stallman shill, go promote leftist licenses somewhere else
Can you elaborate?
Get off my Holla Forums, OSS fag. Nobody likes you.
Go back to /g/ and talk about how you like getting cucked by big corps.
Nice buzzwords, faggot.
i don't care being "cucked" by big corps. i like that my code runs big systems. on the other hand, i like big corps' liberally licensed code. i don't need stallman's leftist agenda to use and develop software.
urbandictionary.com
virtue signalers and larpers are cancer, all that matters is reality.
Classic armchair developer.
How so?
ARE YOU KIDDING ME? IS THIS A JOKE?
He said 250 request/second, and to me that's enough unless you want something popular. I'm not sure what your arguying about.
People were donating for software.
Quit defending the fact that the fat fuck wanted to close up the source code, shithead. He hadn't paid a cent for development, users did. And he believed he was entitled to any rights at all to the software beyond copyleft.
In fact, Josh went above and beyond by catering to their autistic database needs, by spending almost a month forking the framework and fixing the PGSQL binary querying issues for one of their stupid gook employees. Meanwhile, they're still stuck on fucking MySQL thanks to Jim being a fat, greedy fuckhead.
Again: you are defending the fact that a pigfucking subhuman attempted to close up software that people had paid for, when the pigfucker hadn't paid a cent. Hang yourself.
What Hotwheels and Jim were doing was nothing more than attempting to embarass Josh into surrendering control of the copyright, which would've meant it was closed source. Just like Holla Forums is now.
Why replace something that works?
people were donating for software to be used on Holla Forums, nothing more.
...
People were donating for someone to develop software, and the terms of the crowdfundingt were that the software be licensed AGPL.
You can keep sucking the dick of a subhuman pigfucker all you want, he attempted to rip off crowdfunders. It's that simple.
He's a subhuman piece of dog shit, whose company regularly replaces copyright attribution with their own.
They just did it with the Everychan android fork, and rewrote git history to cover it up when it was pointed out, despite the fact that they'd redistributed it like that.
That's illegal.
Because something that works can still suck.
If it is free software you can improve parts that suck instead of reinventing everything. Chance is that other people also want those parts improved too so your patch will be accepted by both maintainer and community.
Look at meguca. Written in go, live updating.
what if the entire thing sucks though and only works because of miracles?
Futabilly, written in PHP, fastest imageboard yet. 6k request/second mainly because it uses static/memory to serve everything.
Problem: requires JS and feature poor
lack of experience in scaling and autism
There are
He was afraid of losing 2ch in court.
look at nntpchan, it's golang.
That's not always possible. For example, it's impossible to get D++ (site-wide delete) to work in a reasonable way on Holla Forums without overhauling the database.
If software sucks enough reinventing everything takes less time than improving the parts that suck.
I do not see why it is not possible, can you please explain it a bit more detail?
Each board gets a different table for posts. If you want to delete a single post that works alright, if you want to delete all posts made from an IP address across a single board that works alright, but if you want to do that across all boards of the site you need to access 11,000+ tables at once and then you run into the join limit because SQL isn't designed for that.
That carries over all the way from Tinyboard. It's a design flaw in Tinyboard, but it wasn't very serious because Tinyboard didn't let users create boards. Having this many boards on a site with centralized board management would be weird.
As far as I know the join limit is too low to fix by deleting dead boards.
The same flaw makes it impossible to view site-wide posting histories, which is a real pain when fighting spam.
It seems fixable.
Create one sitewide table that holds IP and unique-ID pair for every poster.
Set that ID on every post instead of IP.
Create delete on cascade trigger on every table.
Remove ID-IP pair from table and let it all cascade.
I mean "ON DELETE CASCADE".
how fucking retarded.
tamachan won't have this problem. in rust
How the fuck will you keep per board post numbers then?
by not having a retarded database schema
Tell me more about that non-retarded schema, please.
i don't have one yet. but unlike the pajeets that made tinyboard i actually studied computer science. i know what i am doing. in rust
tamachan will truly be the most greatest imageboard.
while you were busy shitposting here i actually wrote my own imageboard in rust
Can you please paste link to it if it is available somewhere online?
Holla Forums/infinity's report queue JSON interface is disturbing.
This is the current (pretty-printed) content of sys.8ch.net
[ { "global" : "0", "board" : "infinity", "post" : "814", "id" : "1198781", "post_content" : { "sticky" : "0", "capcode" : null, "thread" : null, "embed" : null, "bump" : "1465940731", "name" : "Josh", "filehash" : "bc3755843b97a6ace68f6de05b4d4b36\n", "trip" : "!!KPMWYXBaPU", "body" : "jewsh is a fagdsafsdfa", "num_files" : "1", "email" : "", "subject" : "", "body_nomarkup" : "jewsh is a fagdsafsdfa", "files" : "[{\"name\":\"AAAAAAAAAAHHHH.png\",\"type\":\"image\\/png\",\"tmp_name\":\"\\/tmp\\/phpM5J8dO\",\"error\":0,\"size\":170527,\"filename\":\"AAAAAAAAAAHHHH.png\",\"extension\":\"png\",\"file_id\":\"1465940732143\",\"file\":\"1465940732143.png\",\"thumb\":\"1465940732143.jpg\",\"is_an_image\":true,\"hash\":\"bc3755843b97a6ace68f6de05b4d4b36\",\"width\":567,\"height\":462,\"thumbwidth\":255,\"thumbheight\":208,\"file_path\":\"infinity\\/src\\/1465940732143.png\",\"thumb_path\":\"infinity\\/thumb\\/1465940732143.jpg\"}]", "edited_at" : "1475235677", "cycle" : "0", "ip" : "4b13701c75", "time" : "1465940731", "sage" : "0", "id" : "814", "locked" : "0" }, "ip" : "4126376402", "time" : "1476131235", "local" : "1", "reason" : "Example report" }]
The first thing I noticed is that the keys are different from the usual JSON interfaces, like 8ch.net
The file information is given as a string containing JSON-encoded data. There are two layers of JSON on top of it.
What's going on here?
Incompetence.
You have autism
If you want an actual answer. Its because the reports page doesn't use the same json interface as the board page....
So in other words.....
You have autism
Do you mean that it doesn't use the same JSON interface because it doesn't use the same JSON interface, or that it uses double-encoded JSON because it doesn't use the same JSON interface?
Could this be used for an XSS attack? Please tell me there's some kind of sanitation in the report function.
I think that Laravel automatically sanitises strings inserted on layout.
one''two''becomes"reason" : "<i>one</i>two",
It goes through the same process as post content.
Holla Forums doesn't use Laravel. You might be thinking of Infinity Next.
I thought that 8ch uses Infinity Next, I was wrong. Sorry for giving non-usefull info.
Ask Jim
>>>/sudo/
Jim is not Codemonkey.
It's Josh code. I'm surprised he didn't also base64 it.
Nice job trying to blame Josh, I get that it's fun to suck the dick of a fat pigfucker on Holla Forums and make fun of Josh, but this is dud code.
github.com
And just in case retards are unable to read the author information:
While we're on the subject:
github.com
NTTEC/RQI sure is a top-tier software outsourcing firm. Considering they can't properly do json output, multiple times, definitely should give their customers confidence in their obviously high quality work.
It's ok, Josh. Calm down.
Pretty embarassing
Why? They'e equally shit, practically interchangeable.
Btw, you misspelled embarrassing.
Because when you blame someone for shit code I expect you have the competency to take a look at git history.
Determining that it was the dud took literally 30 seconds, despite all of the spaghetti of vichan and the fact that I've never worked on it.
I had no idea this was english class. Thanks for reminding me I need to get spellcheck enabled in Firefox.
So if I said it was Josh-tier code would that make you happy?
No, because at least josh doesn't do raw db calls in the middle of code.
fn hash_ip(ip: String) -> String { let mut hasher = Md5::new(); hasher.input_str("salt"); hasher.input_str(&ip); hasher.result_str()}
hahaha oh wow
whats wrong with it?
...
github.com
installgen2 should kill himself
Using a fixed short public salt. It should be long and randomly generated per instance. That way, having the hash and the source code is not enough to bruteforce the password/IP address.
This is basic stuff. If you don't know it you shouldn't be doing security.
I actually really like endchan but the lack of population and last I checked features over a month back is why I don't really go there
I don't know how it is for high traffic but it seemed nice
pretty much retarded thing to do anyway.
same IP doesn't mean same user
There are too few possible IP values anyway.
Whatever hash you choose to use, it's trivial to just enumerate all IPs.
True, but the lack of site-wide posting histories is a real problem.
Not if you don't know the salt. If the salt is secret and different per instance instead of hardcoded you need access to the server to enumerate them.
Yeah, but if an IP from Russia decides to spam the website with CP, I had rather D++ and delete some post made by an unfortunate Russian user who happened to have that IP before it was taken by the buttnet than take the risk.
Going through the post history first would be a good third option. I occasionally use D+ (local deletion by IP) after scrolling through a spammer's post history to make sure there's nothing else.