- 18847 Views
- 2 replies
- 1 kudos
How to convert dataframe (df), to a excel file that I can share with my colleagues ?
I am working on microsoft azure databrick, I have a final dataframe of shape (3276*23) , I want to share it in form of excel file? How can I do it ( I am using ->df.to_excel('fileOutput.xlsx', sheet_name = 'Sheet1', index = False) , command is runn...
- 18847 Views
- 2 replies
- 1 kudos
- 1 kudos
You could try this way, convert Pyspark Dataframe to Pandas Dataframe then export to excel file.
- 1 kudos
- 15634 Views
- 2 replies
- 1 kudos
Resolved! How do you read an Excel spreadsheet with Databricks
My cluster has Scala 2.12I've installed Maven Library com.crealytics:spark-excel_2.12:0.14.0I get an error java.lang.IllegalStateException: Cannot get a STRING value from a NUMERIC cellwhen trying to execute the following%pythonexcelFileName="/mnt/dl...
- 15634 Views
- 2 replies
- 1 kudos
- 1 kudos
Another way also help for your case is usign Pandas to read excel then convert Pandas Dataframe to Pyspark Dataframe
- 1 kudos
- 8073 Views
- 9 replies
- 3 kudos
Resolved! Databricks Certified Data Engineer Associate Certificate or Badge not received
Hello, I passed Databricks Certified Data Engineer Associate on 28th October 2022, but I haven't received my certificate/badge yet. Please help me with this. Best Regards,Janaka Nawarathna.
- 8073 Views
- 9 replies
- 3 kudos
- 13670 Views
- 3 replies
- 7 kudos
Resolved! Best language to use
Databricks supports SQL, Scala, Python, and R. Is there a most performant language to use on Databricks? I know SQL well but would like to get into one of the other languages and don't know which to focus on.
- 13670 Views
- 3 replies
- 7 kudos
- 1216 Views
- 0 replies
- 2 kudos
Databricks SQL endpoint authentication
Hi all​I have a requirement that goes like this:Users from a particular software that doesn't have out-of-the-box integration with Databricks click on a dashboard, the button click then sends an SQL query​ to Databricks (user gets authenticated in Da...
- 1216 Views
- 0 replies
- 2 kudos
- 7773 Views
- 2 replies
- 12 kudos
Resolved! Add the creation date of a parquet file into a DataFrame
Currently I load multiple parquet file with this code:df = spark.read.parquet("/mnt/dev/bronze/Voucher/*/*")(Inside the Voucher folder, there is one folder by date. Each one containing one parquet file)How can I add a column into this DataFrame, that...
- 7773 Views
- 2 replies
- 12 kudos
- 2556 Views
- 0 replies
- 0 kudos
How are you managing your DLT pipelines to maintain graph readability?
I've been building out a few pipelines in DLT and noticed that the usefulness of the user interface has started breaking down at a glance. I've attached a screenshot of one of my pipelines. It's not very far along and it's already pretty rough. You c...
- 2556 Views
- 0 replies
- 0 kudos
- 6128 Views
- 5 replies
- 18 kudos
Resolved! External KMS integration with Databricks like AWS KMS, Azure Key Vault.
I would like to know how we can integrate Databricks with External KMS providers, like currently it is doing with AWS KMS and Azure Key Valut?Can we import keys from any other KMS?
- 6128 Views
- 5 replies
- 18 kudos
- 18 kudos
@Anubhav Gupta​ Databricks is hosted on the cloud provider which means that all resources used by databricks in the backend are in the cloud. For instance, if you create a cluster, the VMs are launched in AWS as EC2 instances. So the integration of K...
- 18 kudos
- 1902 Views
- 0 replies
- 0 kudos
Migration_pipeline.py failing to get default credentials
cat ~/.databrickscfg looks like this (with the correct token/host values in place of xxxxxx)[DEFAULT]host = xxxxxxtoken = xxxxxxjobs-api-version = 2.0The command I run to start the pipeline with default configured credentials is :sudo python3 migrati...
- 1902 Views
- 0 replies
- 0 kudos
- 4992 Views
- 5 replies
- 2 kudos
Here I am getting this error when i execute left join on two data frame: PythonException: 'pyspark.serializers.SerializationError: Caused by Traceback (most recent call last): going to post full traceback:
I simply do left join on two data frame and both data frame content i was able to print.Here is the code looks like:-df_silver = spark.sql("select ds.PropertyID,\ ds.* from dfsilver as ds LEFT JOIN dfaddmaster as dm \ ...
- 4992 Views
- 5 replies
- 2 kudos
- 2007 Views
- 0 replies
- 6 kudos
Is Azure Database for MySQL - Flexible Server supported as external metastore.
In the docs it's mention that "if you use Azure Database for MySQL as an external metastore, you must change the value of the lower_case_table_names property from 1 (the default) to 2 in the server-side database configuration."However "lower_case_tab...
- 2007 Views
- 0 replies
- 6 kudos
- 958 Views
- 0 replies
- 0 kudos
Why does databricks https://docs.databricks.com/dev-tools/api/latest/scim/scim-users.html#get-users take so long
I've been observing as we added more workspaces and users to those workspaces that fetching users per workspace is now taking 11 minutes or more.Our automation to provision group access is now unacceptably long. I've noted that the UI doesn't suffer...
- 958 Views
- 0 replies
- 0 kudos
- 5725 Views
- 2 replies
- 5 kudos
Resolved! Databricks is automatically creating a _apply_changes_storage table in the database when using apply_changes for Delta Live Tables
Hi there,I am using apply_changes (aka. Delta Live Tables Change Data Capture) and it works fine. However, it seems to automatically create a secondary table in the database metastore called _apply_storage_changes_{tableName}So for every table I use ...
- 5725 Views
- 2 replies
- 5 kudos
- 5 kudos
Hi - Thanks @Hubert Dudek​ I will look into disabling access for the users!
- 5 kudos
- 13545 Views
- 1 replies
- 0 kudos
How to deal with Decimal data type arithmetic operations ?
I am dealing with values ranging from 10^9 to 10^-9 , the sum of values can go up to 10^20 and need accuracy. So I wanted to use Decimal Data type [ Using SQL in Data Science & Engineering workspace]. However, I got to know the peculiar behavior of D...
- 13545 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello Everyone,I understand that there is no best answer for this question. So, I could only do the same thing I found when I surfed the net.The method I found works whenIf you know the range of values you deal with (not just the input data but also ...
- 0 kudos
- 2430 Views
- 2 replies
- 0 kudos
Invalid port error when trying to read from PlanetScale MySQL databse
Using the code below I am attempting to connect to a PlanetScale MySQL database. I get the following error: java.sql.SQLException: error parsing url : Incorrect port value. However the port is the default 3306, and I have used the correct url based o...
- 2430 Views
- 2 replies
- 0 kudos
- 0 kudos
HI @Rachel Cunningham​ ,maybe you can share your `driver` and `url` value (masked)?
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 580 | |
| 349 | |
| 287 |
