cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Mr__E
by Contributor II
  • 5076 Views
  • 5 replies
  • 5 kudos

Resolved! Using shared python wheels for job compute clusters

We have a GitHub workflow that generates a python wheel and uploads to a shared S3 available to our Databricks workspaces. When I install the Python wheel to a normal compute cluster using the path approach, it correctly installs the Python wheel and...

  • 5076 Views
  • 5 replies
  • 5 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 5 kudos

You can mount S3 as a DBFS folder then set that library in "cluster" -> "libraries" tab -> "install new" -> "DBFS" 

  • 5 kudos
4 More Replies
yoniau
by New Contributor II
  • 2652 Views
  • 2 replies
  • 5 kudos

Resolved! Different configurations for same Databricks Runtime version

Hi all,On my DBR installations, s3a scheme is mapped to shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem. On my customer's DBR installations it is mapped to com.databricks.s3a.S3AFileSystem.We both use the same DBR runtime, and none of us has...

  • 2652 Views
  • 2 replies
  • 5 kudos
Latest Reply
Prabakar
Databricks Employee
  • 5 kudos

@Yoni Au​ , If both of you are using the same DBR version, then you should not find any difference. As @Hubert Dudek​ mentioned, there might be some spark configuration change made on one of the clusters. Also, it's worth checking for any cluster sco...

  • 5 kudos
1 More Replies
susan1234567
by New Contributor
  • 1899 Views
  • 1 replies
  • 2 kudos

I cannot access databricks community edition account

Last week, I cannot loginto https://community.cloud.databricks.com/login.html all of a sudden. I tried to set the password, also didn't receive the reset email. It says "Invalid email address or password Note: Emails/usernames are case-sensitive".I e...

  • 1899 Views
  • 1 replies
  • 2 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 2 kudos

@Kaniz Fatma​ can help, additionally you can open ticket here https://help.databricks.com/s/contact-us

  • 2 kudos
Serhii
by Contributor
  • 4165 Views
  • 4 replies
  • 8 kudos

Resolved! init_script error during cluster creation - 101: Network is unreachable

When I run the init_script during cluster creationapt-get update && apt-get install -y ffmpeg libsndfile1-devI get an error in cluster logs E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/o/openal-soft/libopenal1_1.19.1-1_amd64.deb ...

  • 4165 Views
  • 4 replies
  • 8 kudos
Latest Reply
Anonymous
Not applicable
  • 8 kudos

Hi @Sergii Ivakhno​ Could you please check if outbound TCP access for port 80 is allowed in the security group.

  • 8 kudos
3 More Replies
OmanEvisa
by New Contributor
  • 483 Views
  • 0 replies
  • 0 kudos

PROCESS OF APPLYING FOR OMAN E-VISA The Oman e-Visa was initiated in 2018, for making the process easy. Presently, 220 countries in the world are elig...

PROCESS OF APPLYING FOR OMAN E-VISAThe Oman e-Visa was initiated in 2018, for making the process easy. Presently, 220 countries in the world are eligible to apply for Oman e-Visa. Tourists can apply for visas online by submitting the Oman visa applic...

  • 483 Views
  • 0 replies
  • 0 kudos
JBear
by New Contributor III
  • 4192 Views
  • 4 replies
  • 4 kudos

Resolved! Cant find reason but suddenly new Jobs are getting huge job id numbers. example 945270539673815

Created Job ID is suddenly started to make huge numbers, and that is now making problems in Terraform plan, cause int is too big Error: strconv.ParseInt: parsing "945270539673815": value out of rangeIm new on the board and pretty new with Databricks ...

  • 4192 Views
  • 4 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Jere Karhu​ , In case you are using the Job/Run id in API, please be advised that you will need to change the client-side logic to process int64/long and expect a random number. In some cases, you just need to change the declared type in their so...

  • 4 kudos
3 More Replies
Mr__E
by Contributor II
  • 3076 Views
  • 3 replies
  • 3 kudos

Resolved! Importing MongoDB with field names containing spaces

I am currently using a Python notebook with a defined schema to import fairly unstructured documents in MongoDB. Some of these documents have spaces in their field names. I define the schema for the MongoDB PySpark connector like the following:Struct...

  • 3076 Views
  • 3 replies
  • 3 kudos
Latest Reply
Mr__E
Contributor II
  • 3 kudos

Solution: It turns out the issue is not the schema reading in, but the fact that I am writing to Delta tables, which do not currently support spaces. So, I need to transform them prior to dumping. I've been following a pattern of reading in raw data,...

  • 3 kudos
2 More Replies
Krishscientist
by New Contributor III
  • 2655 Views
  • 1 replies
  • 2 kudos

Resolved! Issue when reading .wav file

Hi, I am developing notebook to read .wav files and build Speech Matching Scenario. I have saved files in "/FileStore/tables/doors_and_corners_kid_thats_where_they_get_you.wav".When I wrote code like thisfrom scipy.io import wavfileimport numpy as np...

  • 2655 Views
  • 1 replies
  • 2 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 2 kudos

Try to prefix it with dbfs dbfs:/FileStore or /dbfs/FileStore

  • 2 kudos
Abeeya
by New Contributor II
  • 6245 Views
  • 1 replies
  • 5 kudos

Resolved! How to Overwrite Using pyspark's JDBC without loosing constraints on table columns

Hello,My table has primary key constraint on a perticular column, Im loosing primary key constaint on that column each time I overwrite the table , What Can I do to preserve it? Any Heads up would be appreciatedTried Belowdf.write.option("truncate", ...

  • 6245 Views
  • 1 replies
  • 5 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 5 kudos

@Abeeya .​ , Mode "truncate", is correct to preserve the table. However, when you want to add a new column (mismatched schema), it wants to drop it anyway.

  • 5 kudos
Anonymous
by Not applicable
  • 1315 Views
  • 1 replies
  • 0 kudos

How to resolve Quickbooks error 12007

QuickBooks error 12007 occurs when an update time out occurs. QuickBooks may encounter this error when it cannot connect to the internet if it's unable to access the server. If you want to know its solutions then check out our latest blog on this.

  • 1315 Views
  • 1 replies
  • 0 kudos
Latest Reply
willjoe
New Contributor III
  • 0 kudos

How to Resolve QuickBooks Payroll Update Error 12007?For various possible causes of the QB payroll update error 12007, you need to perform different troubleshooting procedures. Follow the solutions in their given sequence to fix this QuickBooks error...

  • 0 kudos
BasavarajAngadi
by Contributor
  • 5792 Views
  • 11 replies
  • 9 kudos

Resolved! Hi Experts i am new to data bricks and i want to know how data bricks supports real time reporting needs in Business intelligence?

Delta lake have 3 levels to maintain data quality ( bronze , silver and gold tables ) but this supports the reporting and BI solutions how does this supports the streaming analytics ?example : I have an app that loads all the operational data in adls...

  • 5792 Views
  • 11 replies
  • 9 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 9 kudos

@Basavaraj Angadi​ , Why? For simplicity, cost savings etc.You can make it work with 2 'containers' but it is not necessary.

  • 9 kudos
10 More Replies
Anonymous
by Not applicable
  • 3598 Views
  • 2 replies
  • 3 kudos

Resolved! Play the BIG DATA GAME | By Firebolt

https://www.firebolt.io/big-data-gameThe most fun our Bricksters have had in a while at work is thanks to a little BIG DATA thing called The BIG DATA GAME ️This game is the cure for the mid-week blues. The Big Data Game is a simple yet awesome online...

Image Image
  • 3598 Views
  • 2 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

HA! I kept 'dying' there too!

  • 3 kudos
1 More Replies
lukas_vlk
by New Contributor III
  • 8831 Views
  • 3 replies
  • 2 kudos

Resolved! Python Spark Job - error: job failed with error message The output of the notebook is too large.

Hi databricks experts. I am currently facing a problem with a submitted job run on Azure Databricks. Any help on this is very welcome. See below for details:Problem Description:I submitted a python spark task via the databricks cli (v0.16.4) to Azure...

  • 8831 Views
  • 3 replies
  • 2 kudos
Latest Reply
lukas_vlk
New Contributor III
  • 2 kudos

Without any further changes from my side, the error has disappeard since 29.03.2022

  • 2 kudos
2 More Replies
Sudeshna
by New Contributor III
  • 2409 Views
  • 2 replies
  • 3 kudos

How can i pass one of the values from one function to another as an argument in Databricks SQL?

For eg - CREATE OR REPLACE TABLE table2(a INT, b INT);INSERT INTO table2 VALUES (100, 200);CREATE OR REPLACE FUNCTION func1() RETURNS TABLE(a INT, b INT) RETURN (SELECT a+b, a*b from table2);create or replace function calc(p DOUBLE) RETURNS TABLE(val...

image
  • 2409 Views
  • 2 replies
  • 3 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 3 kudos

Yes, it is possible, but with different logic. For scalar, so calc(a) in select calc(a) from func1(); it can only be a query as a table for a scalar is not allowed. So please try something like:CREATE OR REPLACE FUNCTION func_table() RETURNS TABLE(a ...

  • 3 kudos
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels