- 2419 Views
- 1 replies
- 0 kudos
run jar file into databricks
I have created a job which run a jar files, but i have this error NoClassDefFoundError: com/google/cloud/hadoop/gcsio/GoogleCloudStorageFileSystemOptions$TimestampUpdatePredicateCaused by: ClassNotFoundException: com.google.cloud.hadoop.gcsio.GoogleC...
- 2419 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey Aubert, seems you are missing dependent class in jar. Either package the dependent classes in jar or add them into class path.
- 0 kudos
- 10573 Views
- 2 replies
- 0 kudos
File copy from local to dbfs
How to copy a file from local disk to databricks dbfs path. I tried as below but it is throwing me error:code i tried: dbutils.fs.cp("file://c:/user/file.txt",dbfs/data/) and dbutils.fs.cp("file:///c:/user/file.txt",dbfs/data/) error: File not found ...
- 10573 Views
- 2 replies
- 0 kudos
- 0 kudos
I assume you cannot copy files from Local machine to dbfs using dbutils. you can upload files to dbfs using below gui option . Data --> Browse DFS --> Upload
- 0 kudos
- 10190 Views
- 2 replies
- 3 kudos
load files filtered by last_modified in PySpark
Hi, community!How do you think what is the best way to load from Azure ADLS (actually, filesystem doesn't matter) into df onli files modified after some point in time?Is there any function like input_file_name() but for last_modified to use it in a w...
- 10190 Views
- 2 replies
- 3 kudos
- 3 kudos
_metadata will provide file modification timestamp. I tried on dbfs but not sure for ADLS.https://docs.databricks.com/ingestion/file-metadata-column.html
- 3 kudos
- 2499 Views
- 0 replies
- 0 kudos
Bokeh plotting with ipywidgets or in Databricks Dashboard
Databricks' documentation for using bokeh shows using displayHTML, which seems to be incompatible with using ipywidgets, or putting them in Databricks dashboards.I found there is undocumented functionality with plotting interactive altair charts in D...
- 2499 Views
- 0 replies
- 0 kudos
- 10650 Views
- 3 replies
- 3 kudos
Resolved! How to integrate pipeline with Dynatrace ?
Hi guys,Do you know how I integrate pipeline some data to Dynatrace ?Have any idea ?Thank you
- 10650 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @William Scardua​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answe...
- 3 kudos
- 1603 Views
- 0 replies
- 0 kudos
We are on USGov Arizona region with Azure Databricks. We are trying to use Delta Live Tables. We are on Premium pricing tier and I am not able to see the option for delta live tables. Is this supported for UsGov regions?
Delta live option is missing for UsGov regions? Is this going to be available? If not, when will this be available?
- 1603 Views
- 0 replies
- 0 kudos
- 2447 Views
- 0 replies
- 0 kudos
SCD Type-2
Hi All, I have 22 postgress tables and i need to implement SCD type 2 and create azure Databricks pipeline . However my project team doesn't want to use delta tables concept . Have anyone implemented this ? below is how i planned to do try: df_src = ...
- 2447 Views
- 0 replies
- 0 kudos
- 16944 Views
- 2 replies
- 2 kudos
Resolved! can anyone help with Spill Question
Spill occurs as a result of executing various wide transformations. However, diagnosing a spill requires one to proactively look for key indicators.Where in the Spark UI are two of the primary indicators that a partition is spilling to disk?a- Exec...
- 16944 Views
- 2 replies
- 2 kudos
- 2 kudos
@Aviral Bhardwaj​ I feel it is Option e. Stage and executor log files. Consolidated details at the Stage LevelDetails at the task and Executor Level Please let me know if you feel any other option is better.
- 2 kudos
- 3426 Views
- 1 replies
- 3 kudos
lnkd.in
Databricks Auto Loader is an interesting feature that can be used to load data incrementally.✳ It can process new data files as they arrive in the cloud object stores✳ It can be used to ingest JSON, CSV, PARQUET, AVRO, ORC, TEXT and even Binary file ...
- 3426 Views
- 1 replies
- 3 kudos
- 3 kudos
Thanks for sharing
- 3 kudos
- 4607 Views
- 3 replies
- 0 kudos
Resolved! Account Registration Help for Partner-Academy
Hi there,I used my company email to register an account for customer-academy.databricks.com a while ago. And I need to create an account in the partner-academy.databricks.com using the same company email.However, when I tried for partner-academy, it ...
- 4607 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Geunhee Kang​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...
- 0 kudos
- 4066 Views
- 2 replies
- 0 kudos
"Endpoint not found for /2.0/sql/statements/"
Hey everyone! I´m trying to access table row using databricks api. Using Insomnia or postman, to test and the error are the same: { "error_code": "ENDPOINT_NOT_FOUND", "message": "Endpoint not found for /2.0/sql/statements/"}Below is my request:(for ...
- 4066 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Denis Macedo​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 0 kudos
- 2800 Views
- 2 replies
- 1 kudos
I want to import a notebook
Want to import a note book, facing this following error:Importation fails: could not deserialize: Unkown language htm
- 2800 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @BITAIGOT Guy-Serge​ We haven't heard from you since the last response from @Debayan Mukherjee​ ​, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be hel...
- 1 kudos
- 7050 Views
- 3 replies
- 3 kudos
Resolved! Does driver node of job compute have HA?
I would like to confirm and discuss HA mechanism about driver node of job compute. Because we can image driver node just like master node of cluster. In AWS EMR, we can setup 2 master node so that one of master node failed, another master node can re...
- 7050 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Mars Su​ We haven't heard from you since the last response from @Werner Stinckens​ and @karthik p​ ​, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be...
- 3 kudos
- 11387 Views
- 2 replies
- 4 kudos
Resolved! Revoking cluster creation permission for non-admin users in Databricks workspace
I'm currently building a Databricks workspace for my organization and I would like to revoke the cluster creation permission for all users except admins or a specific group (e.g., Data Analysts). Could someone please guide me on how to achieve this?
- 11387 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi @Nicolae Istrate​ We haven't heard from you since the last response from @Werner Stinckens​ , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful ...
- 4 kudos
- 7139 Views
- 2 replies
- 1 kudos
Resolved! How to enable spark.shuffle.compress in spark 3.3.0 or above versions?
when I try to set that I got a following error, appreciate your comments, thanks you in advance.
- 7139 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Ankith Patlolla​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answe...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
