- 6771 Views
- 2 replies
- 4 kudos
Why does Databricks SQL drop ending 0 in decimal data type
All,I have a column, RateAdj that is defined as DECIMAL(15,5) and I can see that the value is 4.00000, but when this gets inserted into my table it shows as just 4.%sql SELECT LTRIM(RTRIM(IFNULL(FORMAT_NUMBER(RateADJ, '0.00000'), '0.00000')))This i...
- 6771 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi @Greg Bradley​ Hope everything is going great.Checking in. If @Uma Maheswara Rao Desula​ answer helped, would you let us know and mark the answer as best? If not, would you be happy to give us more information?Thanks!
- 4 kudos
- 3611 Views
- 1 replies
- 0 kudos
Resolved! Error Using spark.catalog.dropTempView()
I have a set of Spark Dataframes that I convert into Temp Views to run Spark SQL with. Then, I delete them after my logic/use is complete. The delete step throws an odd error that I am not sure how to fix. Looking for some tips on fixing it. As a not...
- 3611 Views
- 1 replies
- 0 kudos
- 0 kudos
spark.sql("DROP TABLE "+prefix_updates) spark.sql("DROP TABLE "+prefix_main)Fixed it for me.
- 0 kudos
- 4961 Views
- 4 replies
- 4 kudos
Passed Data Engineer Associate Exam but received twice the Lakehouse Accreditation
Hi all,today (27/12/22 14:00 Rome Time Zone) I passed the Data Engineer Associate exam, but I received the badge of Lakehouse Fundamentals (second time). My email address is: sisbarra@gmail.comMy company address is: antonio.sisbarra@nttdata.comCan ...
- 4961 Views
- 4 replies
- 4 kudos
- 4 kudos
Hello Antonio,I deleted the badge under: antonio.sisbarra@nttdata.com . You are good to go.Thanks!
- 4 kudos
- 9659 Views
- 9 replies
- 14 kudos
Resolved! Databricks Lakehouse Fundamentals Accreditation V2 badge/certificate not received.
Hello, this is regarding Databricks Lakehouse Fundamentals Accreditation V2. I haven't received my badge/certificate. I also raised a ticket but haven't received any response. My request id is #00248504.​. Kindly help me out with this.
- 9659 Views
- 9 replies
- 14 kudos
- 14 kudos
@Jose Gonzalez​ ,@Vidula Khanna​ ,I also did not received my badge in portalThis is my portal link- https://credentials.databricks.com/profile/aviralbhardwaj143185/walletMy case number is - 00250939Also, my points are not updating in the reward port...
- 14 kudos
- 3966 Views
- 3 replies
- 2 kudos
How do we concatenate some fixed string with a secret value in Spark Config in Databricks Job Cluster?
Hi Team,I am trying to configure access to adls through Service Principal through Spark Config in Databricks job cluster. like, fs.azure.account.oauth2.client.id.<adls_account_name>.dfs.core.windows.net {{secrets/scopeName/clientID}}The above stateme...
- 3966 Views
- 3 replies
- 2 kudos
- 2 kudos
@Kaniz Fatma​ We are blocked on this issue. Can you please look into the thread and give your suggestion to workaround it.
- 2 kudos
- 31329 Views
- 1 replies
- 0 kudos
Resolved! How to show all rows by "DataFrame.show()"?
Hi,DataFrame.show() has a parameter n to set "Number of rows to show".Is there any way to show all rows?
- 31329 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Medothis method will work fine df.show(df.count())
- 0 kudos
- 3202 Views
- 2 replies
- 3 kudos
Resolved! Can't do "Full screen" while taking Databricks Apache Spark developer course.
Hi, I see the option for "Full screen" on bottom right but its disabled/inactive. Attached is the screenshot for same.Please advise as its hard to read or see contents on half screen.Thanks
- 3202 Views
- 2 replies
- 3 kudos
- 3244 Views
- 3 replies
- 3 kudos
Access file location problem
import pandas as pd from apiclient.discovery import build from oauth2client.service_account import ServiceAccountCredentials df = spark.read.json("/FileStore/tables/cert.json") SCOPES = ['https://www.googleapis.com/auth/analytics.readonly'] KEY_FIL...
- 3244 Views
- 3 replies
- 3 kudos
- 3 kudos
Looks like it is because the oauth2client.service_account does not know about DBFS (whereas spark does).Is it an option to manage your secrets in databricks? https://docs.databricks.com/security/secrets/secrets.html
- 3 kudos
- 29378 Views
- 5 replies
- 7 kudos
Resolved! Best Practices for naming Tables and Databases in Databricks
We moved in Databricks since few months from now, and before that we were in SQL Server.So, all our tables and databases follow the "camel case" rule.Apparently, in Databricks the rule is "lower case with underscore".Where can we find an official doc...
- 29378 Views
- 5 replies
- 7 kudos
- 7 kudos
Hi @Salah KHALFALLAH​ , looking at the documentation it appears that Databricks' preferred naming convention is lowercase and underscores as you mentioned.The reason for this is most likely because Databricks uses Hive Metastore, which is case insens...
- 7 kudos
- 3114 Views
- 3 replies
- 3 kudos
Resolved! Why does chaining spark.read from one system/driver and .write to another system/driver take so much longer than doing each piece individually?
i am reading data from IBM DB2 and saving into a MS SQL server (the first step is moving the code itself to databricks, and then we will move the databases to databricks itself). Problem I'm running into is doing something like the below will take > ...
- 3114 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi, it is related to partitioning optimization. By default, the JDBC driver queries the source database with only a single thread. So write was from one partition as one partition was created, so it was using a single core. When you used pandas, it d...
- 3 kudos
- 3095 Views
- 4 replies
- 4 kudos
RStudio on Databricks user experience
Is anybody actually using the RStudio app integration on Databricks? I'm surprised to find so little discussion in this forum. My team has been using it for about 3 months and it seems under-developed.1) No automated backup, you have to do it yoursel...
- 3095 Views
- 4 replies
- 4 kudos
- 4 kudos
@Jonathan Dufault​ Thanks for the response, and glad I'm not alone. My problem (and this is probably just a preference thing) is that the 'reward' of using a full-fledged IDE is huge, compared to bouncing between notebooks in multiple tabs. The integ...
- 4 kudos
- 2307 Views
- 0 replies
- 0 kudos
Singleton Design Principle for pyspark database connector A singleton is a design pattern that ensures that a class has only one instance, and provide...
Singleton Design Principle for pyspark database connectorA singleton is a design pattern that ensures that a class has only one instance, and provides a global access point to that instance. Here is an example of how you could implement a singleton d...
- 2307 Views
- 0 replies
- 0 kudos
- 4464 Views
- 2 replies
- 2 kudos
How to use partial_parse.msgpack with workflow dbt task?
I'm looking for direction on how to get the dbt task in workflows to use the partial_parse.msgpack file to skip parsing files that haven't changed. I'm downloading my artifacts after each run and the partial_parse file is being saved back to adls.Wha...
- 4464 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi, Could you please confirm what will be your expectation and the used case? Do you want the file to be saved somewhere else?
- 2 kudos
- 4967 Views
- 4 replies
- 7 kudos
Resolved! Connecting azure synapse through data bricks note books
Hi All, Happy new year!how can we connect to azure synapse serverless sql pool through databricks notebooks and execute DDLs
- 4967 Views
- 4 replies
- 7 kudos
- 7 kudos
@KVNARK .​ https://joeho.xyz/blog-posts/how-to-connect-to-azure-synapse-in-azure-databricks/
- 7 kudos
- 4612 Views
- 2 replies
- 2 kudos
Read/Write concurrency issue
Hi. I assume that it can be concurrency issue. (a Read thread from Databricks and a Write thread from another system)From the start:I read 12-16 csv files (approximately 250Mb each of them) to dataframe. df = spark.read.option("header", "False").opti...
- 4612 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Anastasiia Polianska​,I agree, it looks like a concurrency issue. Very possibly this concurrency problem will be caused by an erroneous ETAG in the HTTP call to the Azure Storage API (https://azure.microsoft.com/de-de/blog/managing-concurrency-in...
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
