cancel
Turn on suggestions
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
harraz
by • New Contributor III
  • 2402 Views
  • 1 replies
  • 0 kudos

Issues loading files csv files that contain BOM (Byte Order Mark) character

I keep getting and error when creating dataframe or steam from certain CSV files where the header contains BOM (Byte Order Mark) character  This is the error message:AnalysisException: [RequestId=e09c7c8d-2399-4d6a-84ae-216e6a9f8f6e ErrorClass=INVALI...

Data Engineering
Reply
  • 2402 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @mohamed harraz​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 0 kudos
Spark4speed
by • New Contributor
  • 1535 Views
  • 1 replies
  • 0 kudos

CDM connector for Spark can't connect to Azure storage account

Hello,I'm trying to use the CDM connector for Spark, but I can't connecto to the Azure storage account when using the connector. I mounted a container of storage account with a SAS-token. When I'm trying to read CDM data from a (mounted) storage acco...

Data Engineering
Reply
  • 1535 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Martijn de Bruijn​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 0 kudos
nlakshmidevi125
by • New Contributor
  • 3430 Views
  • 2 replies
  • 1 kudos

about .crc file in delta transaction log

why .crc file will create along with delta log files

Data Engineering
Reply
  • 3430 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Lakshay
Databricks Employee
  • 1 kudos

Hi @Lakshmi devi​ , crc file is basically a checksum file that contains the stats for the respective version file. It is used for snapshot verification in the backend.

  • 1 kudos
1 More Replies
Mikki007
by • New Contributor II
  • 6767 Views
  • 2 replies
  • 1 kudos

Resolved! Constructor public org.apache.spark.SparkConf(boolean) is not whitelisted.

My code:from great_expectations.datasource import SparkDFDatasourcefrom pyspark.sql import SparkSessionsession_name = 'mk_spark_session'spark = SparkSession.builder.appName(session_name).getOrCreate()datasource = SparkDFDatasource(spark)query = "SELE...

Data Engineering
Reply
  • 6767 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 1 kudos

Are you using a high concurrency cluster? if you do, please try to run this code on standard cluster

  • 1 kudos
1 More Replies
parthsalvi
by • Contributor
  • 5763 Views
  • 2 replies
  • 2 kudos

getContext() in dbutils.notebook not working in DBR 11.2 10.4 LTS Shared ModeIt's also working in no isolation Mode in DBR 11.2

We are trying to fetch notebook context in our Job logging workflow.   current_context = dbutils.notebook.entry_point.getDbutils().notebook().getContext().toJson()   We were able to access this in DBR 10.4 custom mode but in DBR 10.4 & 11.2 (Shared M...

image
Data Engineering
Reply
  • 5763 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Tjomme
New Contributor III
  • 2 kudos

See also: https://community.databricks.com/s/question/0D58Y00009t95NHSAY/unity-catalog-shared-access-mode-dbutilsnotebookentrypointgetcontext-not-whitelisted

  • 2 kudos
1 More Replies
yopbibo
by • Contributor II
  • 13343 Views
  • 6 replies
  • 0 kudos

Resolved! How to copy the content of a repos, in the workspace/shared, automatically, daily?

How to copy the content of a repos, in the workspace/shared, automatically, daily?Purpose here is to bring some notebooks, in shared, available to all workspace users, without requesting users to use REPOS.

Data Engineering
Reply
  • 13343 Views
  • 6 replies
  • 0 kudos
View Replies
Latest Reply
citizenkrank
New Contributor II
  • 0 kudos

Alternatively, you can schedule a notebook with the following cell:%sh cp -r /Workspace/Repos/username/repo_name /Workspace/SharedPlease note that you'll have to update (i.e. pull) the repo manually if you've updated it somewhere else (although you c...

  • 0 kudos
5 More Replies
gilo12
by • New Contributor III
  • 1046 Views
  • 0 replies
  • 1 kudos

Connect to warehouse vs connect to compute/cluster

When I create a warehouse, there are connection details which I was able to successfully use, but there are also connection details when I navigate in the console to Compute->configuration->Advanced options-> JDBC/ODBC What is the difference between ...

Data Engineering
Reply
  • 1046 Views
  • 0 replies
  • 1 kudos
Kash
by • Contributor III
  • 7715 Views
  • 3 replies
  • 0 kudos

Linear Regression HELP! Pickle + Broadcast Variable Error

Hi there,I need some help with this example. We're trying to create a linearRegression model that can parallelize for thousands of symbols per date. When we run this we get a picklingError Any suggestions would be much appreciated!KError:PicklingErro...

Data Engineering
Reply
  • 7715 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Kash
Contributor III
  • 0 kudos

@Vidula Khanna​ Can you assist?

  • 0 kudos
2 More Replies
BorislavBlagoev
by • Valued Contributor III
  • 12507 Views
  • 3 replies
  • 9 kudos

Resolved! Delta save timestamp as timestamp with time zone

Hello! I have the following problem. I want to save the delta table and that table contains timestamp columns, but when I try to write that table with spark the timestamp columns become timestamp with the time zone. This is a problem in my case becau...

Data Engineering
Reply
  • 12507 Views
  • 3 replies
  • 9 kudos
View Replies
Latest Reply
Bhimaniheet
New Contributor II
  • 9 kudos

Hi @Hubert Dudek​ ,When you have time, I describe my problem. Can you please check?

  • 9 kudos
2 More Replies
mbejarano89
by • New Contributor III
  • 2913 Views
  • 2 replies
  • 0 kudos

Running a K-means (.fit) gives error:Params must be either a param map or a list/tuple of param maps but got %s." % type(params)

 am running a k-means algorithm. My feature are DoubleType and have no nulls, but I get : raise TypeError("Params must be either a param map or a list/tuple of param maps but got %s." % type(params). Anyone have any idea how to solve this?File /datab...

Data Engineering
Reply
  • 2913 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
mbejarano89
New Contributor III
  • 0 kudos

I found the answer just by trying several things, although I do not understand exactly what the problem was. All I had to do was to cache the input data before fitting the model:assemble=VectorAssembler(inputCols=columns_input, outputCol='features')...

  • 0 kudos
1 More Replies
Anonymous
by • Not applicable
  • 2785 Views
  • 4 replies
  • 2 kudos

  Hello Everyone, I am thrilled to announce that we have our 6th winner for the raffle contest - @Yogita Chavan​ . Please join me in congratulating he...

 Hello Everyone,I am thrilled to announce that we have our 6th winner for the raffle contest - @Yogita Chavan​ . Please join me in congratulating her on this remarkable achievement!Your dedication and hard work have paid off, and we are delighted to ...

6
Data Engineering
Reply
  • 2785 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
yogu
Honored Contributor III
  • 2 kudos

@Suteja Kanuri​ i haven't received swag until now as per our last mail i provided size after that I didn't receive any mail for tracking details

  • 2 kudos
3 More Replies
bshirdi
by • New Contributor II
  • 8516 Views
  • 1 replies
  • 2 kudos

Getting HTTP 502 bad gateway error!

Hello all,I am suddenly getting an HTTP 502 and DRIVER_LIBRARY_INSTALLATION_FAILURE error during the Python library installation when the cluster gets initialized. I have around 10 Python packages out of which 2-3, packages always failed to install a...

image.png
Data Engineering
Reply
  • 8516 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Bhargav Shir​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 2 kudos
Priyag1
by • Honored Contributor II
  • 1117 Views
  • 2 replies
  • 10 kudos

How to improve visualization and dasboard concepts?

How to improve visualization and dasboard concepts?

Data Engineering
Reply
  • 1117 Views
  • 2 replies
  • 10 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 10 kudos

Hi @Priyadarshini G​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 10 kudos
1 More Replies
carlosjrestr
by • New Contributor III
  • 3545 Views
  • 1 replies
  • 1 kudos

Does Unity Catalog on Azure require premium blob storage tier?

from the docs I read : Create a storage container where the metastore’s managed table data will be stored.This storage container must be in a Premium performance Azure Data Lake Storage Gen2 account in the same region as the workspaces you want to us...

Data Engineering
Reply
  • 3545 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Carlos Restrepo​ We haven't heard from you since the last response from @Kaniz Fatma​ ​, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to o...

  • 1 kudos
pranavyadavbugy
by • New Contributor II
  • 3022 Views
  • 1 replies
  • 1 kudos

Regarding Discount on certifications for students

Hi team,I'm a student is there any student discounts for students on certification if yes please let me know.Thanks

Data Engineering
Reply
  • 3022 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Pranav Yadav​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 1 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels
Top