- 4396 Views
- 3 replies
- 2 kudos
How do we concatenate some fixed string with a secret value in Spark Config in Databricks Job Cluster?
Hi Team,I am trying to configure access to adls through Service Principal through Spark Config in Databricks job cluster. like, fs.azure.account.oauth2.client.id.<adls_account_name>.dfs.core.windows.net {{secrets/scopeName/clientID}}The above stateme...
- 4396 Views
- 3 replies
- 2 kudos
- 2 kudos
@Kaniz Fatma​ We are blocked on this issue. Can you please look into the thread and give your suggestion to workaround it.
- 2 kudos
- 33054 Views
- 1 replies
- 0 kudos
Resolved! How to show all rows by "DataFrame.show()"?
Hi,DataFrame.show() has a parameter n to set "Number of rows to show".Is there any way to show all rows?
- 33054 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Medothis method will work fine df.show(df.count())
- 0 kudos
- 3719 Views
- 2 replies
- 3 kudos
Resolved! Can't do "Full screen" while taking Databricks Apache Spark developer course.
Hi, I see the option for "Full screen" on bottom right but its disabled/inactive. Attached is the screenshot for same.Please advise as its hard to read or see contents on half screen.Thanks
- 3719 Views
- 2 replies
- 3 kudos
- 3650 Views
- 3 replies
- 3 kudos
Access file location problem
import pandas as pd from apiclient.discovery import build from oauth2client.service_account import ServiceAccountCredentials df = spark.read.json("/FileStore/tables/cert.json") SCOPES = ['https://www.googleapis.com/auth/analytics.readonly'] KEY_FIL...
- 3650 Views
- 3 replies
- 3 kudos
- 3 kudos
Looks like it is because the oauth2client.service_account does not know about DBFS (whereas spark does).Is it an option to manage your secrets in databricks? https://docs.databricks.com/security/secrets/secrets.html
- 3 kudos
- 32676 Views
- 5 replies
- 7 kudos
Resolved! Best Practices for naming Tables and Databases in Databricks
We moved in Databricks since few months from now, and before that we were in SQL Server.So, all our tables and databases follow the "camel case" rule.Apparently, in Databricks the rule is "lower case with underscore".Where can we find an official doc...
- 32676 Views
- 5 replies
- 7 kudos
- 7 kudos
Hi @Salah KHALFALLAH​ , looking at the documentation it appears that Databricks' preferred naming convention is lowercase and underscores as you mentioned.The reason for this is most likely because Databricks uses Hive Metastore, which is case insens...
- 7 kudos
- 3581 Views
- 3 replies
- 3 kudos
Resolved! Why does chaining spark.read from one system/driver and .write to another system/driver take so much longer than doing each piece individually?
i am reading data from IBM DB2 and saving into a MS SQL server (the first step is moving the code itself to databricks, and then we will move the databases to databricks itself). Problem I'm running into is doing something like the below will take > ...
- 3581 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi, it is related to partitioning optimization. By default, the JDBC driver queries the source database with only a single thread. So write was from one partition as one partition was created, so it was using a single core. When you used pandas, it d...
- 3 kudos
- 3621 Views
- 4 replies
- 4 kudos
RStudio on Databricks user experience
Is anybody actually using the RStudio app integration on Databricks? I'm surprised to find so little discussion in this forum. My team has been using it for about 3 months and it seems under-developed.1) No automated backup, you have to do it yoursel...
- 3621 Views
- 4 replies
- 4 kudos
- 4 kudos
@Jonathan Dufault​ Thanks for the response, and glad I'm not alone. My problem (and this is probably just a preference thing) is that the 'reward' of using a full-fledged IDE is huge, compared to bouncing between notebooks in multiple tabs. The integ...
- 4 kudos
- 2536 Views
- 0 replies
- 0 kudos
Singleton Design Principle for pyspark database connector A singleton is a design pattern that ensures that a class has only one instance, and provide...
Singleton Design Principle for pyspark database connectorA singleton is a design pattern that ensures that a class has only one instance, and provides a global access point to that instance. Here is an example of how you could implement a singleton d...
- 2536 Views
- 0 replies
- 0 kudos
- 4899 Views
- 2 replies
- 2 kudos
How to use partial_parse.msgpack with workflow dbt task?
I'm looking for direction on how to get the dbt task in workflows to use the partial_parse.msgpack file to skip parsing files that haven't changed. I'm downloading my artifacts after each run and the partial_parse file is being saved back to adls.Wha...
- 4899 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi, Could you please confirm what will be your expectation and the used case? Do you want the file to be saved somewhere else?
- 2 kudos
- 5555 Views
- 4 replies
- 7 kudos
Resolved! Connecting azure synapse through data bricks note books
Hi All, Happy new year!how can we connect to azure synapse serverless sql pool through databricks notebooks and execute DDLs
- 5555 Views
- 4 replies
- 7 kudos
- 7 kudos
@KVNARK .​ https://joeho.xyz/blog-posts/how-to-connect-to-azure-synapse-in-azure-databricks/
- 7 kudos
- 5148 Views
- 2 replies
- 2 kudos
Read/Write concurrency issue
Hi. I assume that it can be concurrency issue. (a Read thread from Databricks and a Write thread from another system)From the start:I read 12-16 csv files (approximately 250Mb each of them) to dataframe. df = spark.read.option("header", "False").opti...
- 5148 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Anastasiia Polianska​,I agree, it looks like a concurrency issue. Very possibly this concurrency problem will be caused by an erroneous ETAG in the HTTP call to the Azure Storage API (https://azure.microsoft.com/de-de/blog/managing-concurrency-in...
- 2 kudos
- 6744 Views
- 2 replies
- 4 kudos
- 6744 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi @maddy v​ ,I recommend that you use the Databricks SQL module for this type of reports and email alerts. It is a very interesting module with multiple options for your use case.https://learn.microsoft.com/en-us/azure/databricks/sql/user/dashboards...
- 4 kudos
- 1783 Views
- 1 replies
- 3 kudos
Spark SQL & Spark ML
I am using Spark SQL to import their data into a machine learning pipeline. Once data is imported I want performs machine learning tasks using Spark ML. So I should use what compute tools is best suited for this use case? Please help me!!! Thank you ...
- 1783 Views
- 1 replies
- 3 kudos
- 3 kudos
Hi, please refer https://docs.databricks.com/machine-learning/index.html, please let us know if this helps.
- 3 kudos
- 1852 Views
- 1 replies
- 2 kudos
I am using Spark SQL to import their data into a machine learning pipeline. Once data is imported I want performs machine learning tasks using Spark...
I am using Spark SQL to import their data into a machine learning pipeline. Once data is imported I want performs machine learning tasks using Spark ML. So I should use what compute tools is best suited for this use case? Please help me!!! Thank y...
- 1852 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi, please refer https://docs.databricks.com/machine-learning/index.html, please let us know if this helps.
- 2 kudos
- 2046 Views
- 1 replies
- 2 kudos
How to Enable Files in Repos in DBR 7.3 LTS ML ?
we need a custom version of a GPU cluster with following requirements for a certain project:Ubuntu 18.04Cuda 10.1.Tesla T4 GPU.Availability of /Workspace/Repos folder.All of these requirements are available with DBR ML 7.3 LTS. But one critical compo...
- 2046 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi, To work with non-notebook files in Databricks Repos, you must be running Databricks Runtime 8.4 or above.https://docs.databricks.com/files/workspace.html#configure-support-for-workspace-files
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
