- 10856 Views
- 3 replies
- 0 kudos
Resolved! What is the best practice for managing different environments (Staging vs Production) on Databricks?
Should we create separate workspaces for Dev/Test/Prod ? Or should we have 1 workspace and create separate folders for Dev/Test/Prod?
- 10856 Views
- 3 replies
- 0 kudos
- 0 kudos
as per my previous experience, its always good to have different workspaces for different envs, its easy to maintain and helps better with CICD pipeline as well, because lot of organizations provide deployment access to Developers in Dev env but not ...
- 0 kudos
- 1688 Views
- 1 replies
- 0 kudos
run jar file into databricks
I have created a job which run a jar files, but i have this error NoClassDefFoundError: com/google/cloud/hadoop/gcsio/GoogleCloudStorageFileSystemOptions$TimestampUpdatePredicateCaused by: ClassNotFoundException: com.google.cloud.hadoop.gcsio.GoogleC...
- 1688 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey Aubert, seems you are missing dependent class in jar. Either package the dependent classes in jar or add them into class path.
- 0 kudos
- 8211 Views
- 2 replies
- 0 kudos
File copy from local to dbfs
How to copy a file from local disk to databricks dbfs path. I tried as below but it is throwing me error:code i tried: dbutils.fs.cp("file://c:/user/file.txt",dbfs/data/) and dbutils.fs.cp("file:///c:/user/file.txt",dbfs/data/) error: File not found ...
- 8211 Views
- 2 replies
- 0 kudos
- 0 kudos
I assume you cannot copy files from Local machine to dbfs using dbutils. you can upload files to dbfs using below gui option . Data --> Browse DFS --> Upload
- 0 kudos
- 7703 Views
- 2 replies
- 3 kudos
load files filtered by last_modified in PySpark
Hi, community!How do you think what is the best way to load from Azure ADLS (actually, filesystem doesn't matter) into df onli files modified after some point in time?Is there any function like input_file_name() but for last_modified to use it in a w...
- 7703 Views
- 2 replies
- 3 kudos
- 3 kudos
_metadata will provide file modification timestamp. I tried on dbfs but not sure for ADLS.https://docs.databricks.com/ingestion/file-metadata-column.html
- 3 kudos
- 1764 Views
- 0 replies
- 0 kudos
Bokeh plotting with ipywidgets or in Databricks Dashboard
Databricks' documentation for using bokeh shows using displayHTML, which seems to be incompatible with using ipywidgets, or putting them in Databricks dashboards.I found there is undocumented functionality with plotting interactive altair charts in D...
- 1764 Views
- 0 replies
- 0 kudos
- 6520 Views
- 3 replies
- 1 kudos
Resolved! How to integrate pipeline with Dynatrace ?
Hi guys,Do you know how I integrate pipeline some data to Dynatrace ?Have any idea ?Thank you
- 6520 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @William Scardua​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answe...
- 1 kudos
- 688 Views
- 0 replies
- 0 kudos
We are on USGov Arizona region with Azure Databricks. We are trying to use Delta Live Tables. We are on Premium pricing tier and I am not able to see the option for delta live tables. Is this supported for UsGov regions?
Delta live option is missing for UsGov regions? Is this going to be available? If not, when will this be available?
- 688 Views
- 0 replies
- 0 kudos
- 1586 Views
- 0 replies
- 0 kudos
SCD Type-2
Hi All, I have 22 postgress tables and i need to implement SCD type 2 and create azure Databricks pipeline . However my project team doesn't want to use delta tables concept . Have anyone implemented this ? below is how i planned to do try: df_src = ...
- 1586 Views
- 0 replies
- 0 kudos
- 10753 Views
- 2 replies
- 2 kudos
Resolved! can anyone help with Spill Question
Spill occurs as a result of executing various wide transformations. However, diagnosing a spill requires one to proactively look for key indicators.Where in the Spark UI are two of the primary indicators that a partition is spilling to disk?a- Exec...
- 10753 Views
- 2 replies
- 2 kudos
- 2 kudos
@Aviral Bhardwaj​ I feel it is Option e. Stage and executor log files. Consolidated details at the Stage LevelDetails at the task and Executor Level Please let me know if you feel any other option is better.
- 2 kudos
- 1933 Views
- 1 replies
- 3 kudos
lnkd.in
Databricks Auto Loader is an interesting feature that can be used to load data incrementally.✳ It can process new data files as they arrive in the cloud object stores✳ It can be used to ingest JSON, CSV, PARQUET, AVRO, ORC, TEXT and even Binary file ...
- 1933 Views
- 1 replies
- 3 kudos
- 3 kudos
Thanks for sharing
- 3 kudos
- 2806 Views
- 3 replies
- 0 kudos
Resolved! Account Registration Help for Partner-Academy
Hi there,I used my company email to register an account for customer-academy.databricks.com a while ago. And I need to create an account in the partner-academy.databricks.com using the same company email.However, when I tried for partner-academy, it ...
- 2806 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Geunhee Kang​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...
- 0 kudos
- 2851 Views
- 2 replies
- 0 kudos
"Endpoint not found for /2.0/sql/statements/"
Hey everyone! I´m trying to access table row using databricks api. Using Insomnia or postman, to test and the error are the same: { "error_code": "ENDPOINT_NOT_FOUND", "message": "Endpoint not found for /2.0/sql/statements/"}Below is my request:(for ...
- 2851 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Denis Macedo​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 0 kudos
- 1824 Views
- 2 replies
- 1 kudos
I want to import a notebook
Want to import a note book, facing this following error:Importation fails: could not deserialize: Unkown language htm
- 1824 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @BITAIGOT Guy-Serge​ We haven't heard from you since the last response from @Debayan Mukherjee​ ​, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be hel...
- 1 kudos
- 4630 Views
- 3 replies
- 3 kudos
Resolved! Does driver node of job compute have HA?
I would like to confirm and discuss HA mechanism about driver node of job compute. Because we can image driver node just like master node of cluster. In AWS EMR, we can setup 2 master node so that one of master node failed, another master node can re...
- 4630 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Mars Su​ We haven't heard from you since the last response from @Werner Stinckens​ and @karthik p​ ​, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be...
- 3 kudos
- 7953 Views
- 2 replies
- 4 kudos
Resolved! Revoking cluster creation permission for non-admin users in Databricks workspace
I'm currently building a Databricks workspace for my organization and I would like to revoke the cluster creation permission for all users except admins or a specific group (e.g., Data Analysts). Could someone please guide me on how to achieve this?
- 7953 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi @Nicolae Istrate​ We haven't heard from you since the last response from @Werner Stinckens​ , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful ...
- 4 kudos
| User | Count |
|---|---|
| 1612 | |
| 769 | |
| 348 | |
| 286 | |
| 252 |
