- 6814 Views
- 9 replies
- 0 kudos
Resolved! Query AWS Redshift from Databricks SQL
Greetings,In the documentation for Databricks SQL it states that it supports JDBC connections, however when connecting to AWS Redshift via the built in PostgreSQL driver ("CREATE TABLE sample USING JDBC" and "jdbc://postgresql:/..." URI) I'm getting ...
- 6814 Views
- 9 replies
- 0 kudos
- 0 kudos
@Bilal Aslam anytime! Is there a place where customers could follow the timeline when such features are introduced?
- 0 kudos
- 6648 Views
- 7 replies
- 3 kudos
Resolved! Issue with AWS Glue metacatalogue and DBR 9.1 ... 10.1
I have a simple SparkSQL Select statementoffers_df = (spark.sql(""" SELECT * FROM delta.`{}` """.format(TABLE_LOCATION)))It runs under DBR 9.0 and previous. When changing the DBR to 9.1 ...10.1 I get an exceptionorg.apache.spark.SparkException: Una...
- 6648 Views
- 7 replies
- 3 kudos
- 3 kudos
Retest in DBR 10.3 beta SUCCESSFUL.Problem does not occur in DBR 10.3 beta
- 3 kudos
- 38616 Views
- 4 replies
- 4 kudos
Resolved! Spark is not able to resolve the columns correctly when joins data frames
Hello all, I m using pyspark ( python 3.8) over spark3.0 on Databricks. When running this DataFrame join:next_df = days_currencies_matrix.alias('a').join( data_to_merge.alias('b') , [ days_currencies_matrix.dt == data_to_merge.RATE_DATE, days...
- 38616 Views
- 4 replies
- 4 kudos
- 4 kudos
@Alessio Palma - Howdy! My name is Piper, and I'm a moderator for the community. Would you be happy to mark whichever answer solved your issue so other members may find the solution more quickly?
- 4 kudos
- 6645 Views
- 6 replies
- 4 kudos
Resolved! SQL Analytics: Is it possible to configure the default number format
HiWhen querying an integer value, the default format is '0.0' which results in an integer value 202111, displayed as 202,111. I can resolve the issue by updating the visualisation or using formatnumber in my query, however I would like to set a defa...
- 6645 Views
- 6 replies
- 4 kudos
- 4 kudos
@Bilal Haniff - Would you be happy to mark whichever answer helped you the most as best? That helps others find solutions more quickly.
- 4 kudos
- 2187 Views
- 1 replies
- 15 kudos
Resolved! Write to Azure Delta Lake - optimization request
Databricks/Delta team could optimize some commands which writes to Azure Blob Storage as Azure display that message:
- 2187 Views
- 1 replies
- 15 kudos
- 15 kudos
Hey there. Thank you for your suggestion. I'll pass this up to the team.
- 15 kudos
- 4457 Views
- 2 replies
- 4 kudos
Resolved! How to show the properties of the folders/files from DBFS ?
Hello,How to show the properties of the folders/files from DBFS ?Currently i am using this command :display(dbutils.fs.ls("dbfs:/"))But it only shows :pathnamesizeHow to show these properties ? : CreatedBy (Name)CreatedOn (Date)ModifiedBy (Name)Modi...
- 4457 Views
- 2 replies
- 4 kudos
- 4 kudos
Only one idea is to use %sh magic command but there is no name (just root)
- 4 kudos
- 907 Views
- 0 replies
- 0 kudos
Not having a standard height becomes quite a challenge for many growing adolescents, which is said to be a condition of idiopathic short stature. So, ...
Not having a standard height becomes quite a challenge for many growing adolescents, which is said to be a condition of idiopathic short stature. So, while we can think it will be “ok” with time, we can’t ignore the elephant in the room. To diagnose ...
- 907 Views
- 0 replies
- 0 kudos
- 1651 Views
- 0 replies
- 0 kudos
Inventory Management System Handling the inventory and keeping track of everything is the most difficult, yet important aspect of leading a business...
Inventory Management System Handling the inventory and keeping track of everything is the most difficult, yet important aspect of leading a business. Inventory never stays the same. Due to sales, return, purchases, etc your business’ inventory is alw...
- 1651 Views
- 0 replies
- 0 kudos
- 2945 Views
- 1 replies
- 3 kudos
Resolved! Can I use Databricks to join data from S3 and Postgres using SQL?
Hello, I'm very much new to Databricks and I'm finding it hard if it's right solution for our needs.Requirement:We have multiple data sources spread across AWS S3 and Postgres. We need a common SQL endpoint that can be used to write queries to join d...
- 2945 Views
- 1 replies
- 3 kudos
- 3 kudos
Yes you can. You can ETL to data lake storage register your tables to metastore and register your SELECT with JOINS as VIEW or even better create additionally jobs and store your JOINED table. From BI you can connect to databricks sql or to data lake...
- 3 kudos
- 4504 Views
- 4 replies
- 2 kudos
Resolved! Value Labels fail to display in Databricks notebook but they are displayed ok in Jupyter
import matplotlib.pyplot as pltimport seaborn as snsimport pandas as pdimport numpy as npprob = np.random.rand(7) + 0.1prob /= prob.sum()df = pd.DataFrame({'department': np.random.choice(['helium', 'neon', 'argon', 'krypton', 'xenon', 'radon', 'ogane...
- 4504 Views
- 4 replies
- 2 kudos
- 2 kudos
@Maria Bruevich - Do either of these answers help? If yes, would you be happy to mark one as best so that other members can find the solution more quickly?
- 2 kudos
- 7517 Views
- 4 replies
- 1 kudos
Resolved! Saprk UI not showing any running tasks
HI,I am running a Notebook job calling a JAR code (application code implmented in C#). in the Spark UI page for almost 2 hrs, it'w not showing any tasks and even the CPU usage is below 20%, memory usage is very small. Before this 2 hr window it shows...
- 7517 Views
- 4 replies
- 1 kudos
- 3403 Views
- 4 replies
- 0 kudos
Resolved! Customer deployment
Hi,I have a bunch of scripts in Databricks that perform a decent amount of data-wrangling. All of these scripts contain sensitive information and I have no intention of making them public.I would like to provide a service to my customers - so they ca...
- 3403 Views
- 4 replies
- 0 kudos
- 0 kudos
@Tamas D I understood your concern. For cluster creation in different subscription I think that's by design at this moment. But I would like to request you to add your use case to https://feedback.azure.com/d365community/forum/2efba7dc-ef24-ec11-b6...
- 0 kudos
- 1945 Views
- 2 replies
- 0 kudos
Hi all, I'm having some trouble with my Certification Transcript in the Academy Portal. I've passed "Databricks Certified Associate Devel...
Hi all,I'm having some trouble with my Certification Transcript in the Academy Portal. I've passed "Databricks Certified Associate Developer for Apache Spark 3.0" last year and everything seemed fine (apart from the fact that I've been issued two sep...
- 1945 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey @Piper Wilson ! Thank you for your response. Unfortunately, I already created a support ticket through the address provided in this post you mentioned. And I got a 'case closed' e-mail after over two weeks with no response and no fix (certificat...
- 0 kudos
- 4773 Views
- 3 replies
- 2 kudos
Resolved! Pricing Spot Instance vs New Job Cluster
We are running multiple Databricks job via ADF. I was wondering which option out of the below is a cheaper route for databricks notebook processing from ADF. When I create a ADF linked service, which should I use to lower my cost.New Job Cluster opti...
- 4773 Views
- 3 replies
- 2 kudos
- 2 kudos
the instance pool will be cheaper if you use spot instances. But only if you size your instance pool correctly. (number of workers and scale down time)AFAIK you cannot use spot instances for job clusters in ADF
- 2 kudos
- 7855 Views
- 5 replies
- 2 kudos
Resolved! Pass variable values from one task to another
I created a Databricks job with multiple tasks. Is there a way to pass variable values from one task to another. For example, if I have tasks A and B as Databricks notebooks. Can I create a variable (e.g. x) in notebook A and later use that value in ...
- 7855 Views
- 5 replies
- 2 kudos
- 2 kudos
you could also consider using an orchestration tool like Data Factory (Azure) or Glue (AWS). there you can inject and use parameters from notebooks.The job scheduling of databricks also has the possibility to add parameters, but I do not know if yo...
- 2 kudos
| User | Count |
|---|---|
| 1629 | |
| 790 | |
| 511 | |
| 349 | |
| 287 |
