- 8950 Views
- 2 replies
- 3 kudos
Resolved! Best Data Model for moving from DW to Delta lake
I’m curious what Databricks recommends how we model the data. Do they recommend that the data be in 3rd normal form (3NF). Or should be it be dimensionally modeled (facts and dimensions)
- 8950 Views
- 2 replies
- 3 kudos
- 3 kudos
It all depends on the use case.3NF is ideal for transactional systems. So for a data warehouse/lakehouse that might not be ideal.However there certainly are cases where it is interesting.Star schema's are def still relevant, BUT with the processing p...
- 3 kudos
- 10154 Views
- 5 replies
- 3 kudos
Resolved! What happens to the clusters whose jobs are canceled or terminated due to failures? (Jobs triggered through Job API2.1 using runs/submit)
I am using Databeicks Job Api 2.1 to trigger and run my jobs. "jobs/runs/submit" this API helps in starting the cluster, as well as create the job and run it. This API works great for normal jobs as it also cleans the cluster once job is finished suc...
- 10154 Views
- 5 replies
- 3 kudos
- 3 kudos
@Junee, Anytime! It is crisply mentioned in the doc too. https://docs.databricks.com/clusters/index.html
- 3 kudos
- 34867 Views
- 12 replies
- 10 kudos
Resolved! Databricks rbase container: Rstudio doesn´t work
Hello, How are you? I hope you are doing well!I´m trying to use a databrick´s image (link: containers/ubuntu/R at master · databricks/containers (github.com)) to run a container when starting a cluster. I need that Rstudio is installed on the contain...
- 34867 Views
- 12 replies
- 10 kudos
- 10 kudos
If the issue is resolved would you be happy to mark the answer as best so that others can quickly find the solution in the future.
- 10 kudos
- 15089 Views
- 7 replies
- 2 kudos
Resolved! Can I disable the workspace directory for specific user groups?
We want to use the REPO directory in our production environment only and have a dev environment with less restrictions. If I use the checkbox on the group admin screen to disable workspace access, it locks out the entire Data Engineering section.
- 15089 Views
- 7 replies
- 2 kudos
- 2 kudos
So I found a way to get 85% of the way there:1) Disable workspace access for the users group.2) Create a new group or use another group that you created for the next step.3) Go to the workspace and right click on whitespace in the root directory.4) A...
- 2 kudos
- 14427 Views
- 4 replies
- 5 kudos
Resolved! Is it possible to show multiple cmd output in a dashboard?
I have a loop that outputs a dataframe for values in a list; basically a loop. I can create a dashboard if there is only one df but in the loop, I'm only able to see the charts in the notebook if I switch the view to charts not in the dashboard. In t...
- 14427 Views
- 4 replies
- 5 kudos
- 5 kudos
If you want to be able to easily run and kill multiple process with ctrl-c, this is my favorite method: spawn multiple background processes in a (…) subshell, and trap SIGINT to execute kill 0, which will kill everything spawned in the subshell group...
- 5 kudos
- 9845 Views
- 2 replies
- 5 kudos
Resolved! %pip/%conda doesn't work with encrypted clusters starting DBR 9.x
While trying to use the magic command %pip/%conda with DBR 9.x or above it fails with the following error: %pip install numpy org.apache.spark.SparkException: %pip/%conda commands use unencrypted NFS and are disabled by default when SSL encryption ...
- 9845 Views
- 2 replies
- 5 kudos
- 5 kudos
If you are not aware of the traffic encryption between cluster worker nodes, you can refer to the below link.https://docs.microsoft.com/en-us/azure/databricks/security/encryption/encrypt-otw
- 5 kudos
- 15209 Views
- 5 replies
- 7 kudos
Resolved! Best mechanism to logging the notebook run/metadata and error details
Hi,How we can integrate log analytics with databricks to log notebook run details and code validations.Thank you
- 15209 Views
- 5 replies
- 7 kudos
- 7 kudos
I think you are looking for sending application logs.I'd use Log4j as this is already used by Databricks.The link does not use notebooks but it should work in notebooks too.
- 7 kudos
- 1519 Views
- 1 replies
- 0 kudos
An set up corporation’s image is the entirety. The right campaign strategies can make or ruin a organization’s brand image.business consultant Through...
An set up corporation’s image is the entirety. The right campaign strategies can make or ruin a organization’s brand image.business consultant Through digital advertising and marketing, powerful campaigns may be designed and the scope fixing any glit...
- 1519 Views
- 1 replies
- 0 kudos
- 0 kudos
CP500 is known as “Advance payment of income tax in installments” or “NotisBayaranAnsuran”. CP500 is a tax payment scheme designed by the IRM / LHDN for taxpayers to report their other forms of income, such as rental income, royalties, or other busin...
- 0 kudos
- 1163 Views
- 0 replies
- 0 kudos
Find a local spokesperson for advice. Ask about their career path, how did they "get here"?Read books about speaking and writing.Analyze fa...
Find a local spokesperson for advice. Ask about their career path, how did they "get here"?Read books about speaking and writing.Analyze famous speeches text to speech software for yourself and do not rely on books that tell you the "why" and "how" o...
- 1163 Views
- 0 replies
- 0 kudos
- 15736 Views
- 2 replies
- 4 kudos
Resolved! Parameterize a notebook
I was wondering if there's a way to parameterize a notebook similar to how the Papermill library allows you to parameterize Jupyter notebooks?
- 15736 Views
- 2 replies
- 4 kudos
- 4 kudos
you can try with widgets:https://docs.databricks.com/notebooks/widgets.htmlNot exactly the same as Papermill but it works fine. You can pass the values from your job orchestration tool into a widget so the notebook gets executed with the correct val...
- 4 kudos
- 8889 Views
- 4 replies
- 1 kudos
- 8889 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @Vamsee krishna kanth Arcot Yes, currently you will have to download the JDBC from https://databricks.com/spark/jdbc-drivers-download and connect from other applications with JDBC URL just like you mentioned in your example. There is an internal ...
- 1 kudos
- 12221 Views
- 5 replies
- 4 kudos
Resolved! Programatic access to Files in Repos
Hi, we are testing the new Files support in Databricks repos. Is there a way how to programmatically read notebooks?Thanks
- 12221 Views
- 5 replies
- 4 kudos
- 4 kudos
Hi @Jiri Koutny these files anyway should be synced to your remote repository (git, bitbucket, GitLab etc). The APIs from version control tools Git API for example might help you achieve what you want. https://stackoverflow.com/questions/38491722/r...
- 4 kudos
- 1902 Views
- 1 replies
- 0 kudos
Is there an equivalent of the %debug from Jupyter notebooks in Databricks notebooks for debugging python notebooks?
Is there an equivalent of the %debug from Jupyter notebooks in Databricks notebooks for debugging python notebooks?
- 1902 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Nathan Tong You can go through the 2 articles below that I found online for Debugging in Databricks.1. 7 Tips to Debug Apache Spark Code Faster with Databricks 2. Easier Spark Code Debugging
- 0 kudos
- 4652 Views
- 4 replies
- 0 kudos
I am not able to create a cluster
Hi,I am new on the Databricks platform, few weeks before I created a community version and it was working perfectly till 2 days before, now I can not create a cluster anymore, after few minutes it time out whenever I am trying to create a new cluster...
- 4652 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Ashwinkumar Jayakumar and @Prabakar Ammeappin , I am facing the same issue for 3-4 days.Is there something wrong with Community Edition right now or does my account facing some issues?
- 0 kudos
- 4449 Views
- 2 replies
- 0 kudos
Resolved! External metastore version
I am setting up an external metastore to connect my Databricks cluster. Which is the preferred and recommended Hive metastore version? Also are there any preference or recommendations on the database instance size/type
- 4449 Views
- 2 replies
- 0 kudos
- 0 kudos
@Harikrishnan Kunhumveettil we use databricks runtime 7.3LTS and 9.1LTS. And external hive metastore hosted on azue sql db. Using global init script I have set spark.sql.hive.metastore.version 2.3.7 and downloaded spark.sql.hive.metastore.jars f...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
