Not sure whether better do ask this in an Azure or Spark subject, but I thought I might get responses appropriate to our use cases here. We have Azure Databricks set up and working, and not had any problems following along the tutorials, but I don't...
Hey guys, I am looking to create a real-time analytics application and I am pretty new to Data engineering. Any advice here would be appreciated. So I have been l appvalleyooking into spark streaming for my transformation process, so th tutuappe ove...
like in pandas I usually do df['columnname'].unique()
Hi, this worked for me. distinct_ids = [x.id for x in data.select('id').distinct().collect()]
Hi guys, im new using databricks and i have a challenge in my new work. routerlogin I need to access to one the database (the database is on DBFS) result of some ETLS trough any service, can be ODBC or by some API. I need to connect there because I...
Use the Simba ODBC connector: https://docs.databricks.com/integrations/bi/jdbc-odbc-bi.html
I have been trying to create a new cluster to use and multiple attempts have gotten stuck in pending: "Finding instances for new nodes, acquiring more instances if necessary" until they time out. Up to today I have had no problems creating clusters ...
This is typically a cloud provider issue. You can file a support ticket if the issue persists.
I’m trying to use LSH approxSimilarityJoin on a dataset with ~25k 300-d vectors of floats. It gets stuck and eventually fails with ’Slave lost’ error. The size of cluster and memory are likely not a problem, the failure happens even with 16 nodes, 1...
Use a PandasUDF with Arrow enabled. They are improved in Spark 3, but you can use them in Spark 2.4.5.
Hello people,I'm trying to build a facial recognition application, and I have a working API, that takes in an image of a face and spits out a vector that encodes it. I need to run this on a million faces, store them in a db and when the system goes o...
You could do this with Spark storing in parquet/Delta. For each face you would write out a record with a column for metadata, a column for the encoded vector array, and other columns for hashing. You could use a PandasUDF to do the distributed dista...
Can we use databricks delta lake as a data warehouse kind of thing where business analysts can explore data according to their needs ? Delta lake provides following features which I think supports this idea support to sql syntaxprovide ACID guarante...
@austiamel47, Yes, you can certainly do this. Delta Lake is designed to be competitive with traditional data warehouses and with some tuning can power low-latency dashboards.https://databricks.com/glossary/data-lakehouse
Yes. A new workspace would need to be deployed because Azure allows people to change the vnet cidr but it requires you to remove all the vnet resources first. This includes the Databricks deployment, therefore, this is an Azure restriction on how VNE...
Hi, we need create an interactive map from ipyleaflet library and this use jupyterlab extensionjupyter labextension install @jupyter-widgets/jupyterlab-manager jupyter-leafletWe achieved to show with displayHTML but we lose the widget events
The Spark driver / executors have callbacks to an external service. I need to find out the IP ranges to be locked down for the firewall on the external service. Here is what I found from Azure (https://www.microsoft.com/en-us/download/details.aspx?...
Hello, Relatively new to Databricks and I've been using the Community Edition for a little bit now. I've recently been having more and more issues with my clusters pending until they time out before ever starting up. I've seen a few other posts here...
Test Latesst Question
test comment
Hi everybody! Is it possible to use one job cluster for multiple tasks using new task orchestration tool? If so then please tell how. If it's impossible then this new tool looks useless Regards, Daniil.
Databricks team, are there any news about it?
The the regular version of databricks, the dbfs is mounted at /dbfs. This does not seem to be the case with community edition. I am seeking more details.
If you are using DBR 7.x. or newer versions the dbfs is disabled. You have to use dbutils.fs.cp commands as a workaround.
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
